\subsubsection{Proof of Theorem~\ref{OracleInSta}}

\subsubsection{Proof of Theorem~\ref{OracleInSta}}

\begin{proof}
The proof approach is based on   Taylor's theorem and  the definition of stationary points.

We start by defining some notations:
We use the notation $\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{A}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}~\deq~\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top[\textcolor{\draftcolor}{\ensuremath{\Theta}},\textcolor{\draftcolor}{A}]\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}$ to generate an extended network indexed by $(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{A}}$) with
 $\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}\deq(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}^\top,\textcolor{\draftcolor}{\tilde{\datain}}^\top)^\top\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{w-1}}$,  $\textcolor{\draftcolor}{\tilde{\datain}}\sim\textcolor{\draftcolor}{\mathcal{N}(\boldsymbol{0},I_{\textcolor{\draftcolor}{w-1}\times\textcolor{\draftcolor}{w-1}})}$,
 and  $\textcolor{\draftcolor}{A}=[\textcolor{\draftcolor}{\boldsymbol{v}}_1,\dots,\textcolor{\draftcolor}{\boldsymbol{v}}_{\textcolor{\draftcolor}{\ensuremath{w}}-1}]\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times\textcolor{\draftcolor}{w-1}}$, with $\textcolor{\draftcolor}{\boldsymbol{v}}_1,\dots,\textcolor{\draftcolor}{\boldsymbol{v}}_{\textcolor{\draftcolor}{\ensuremath{w}}-1}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$, is a matrix whose columns are basis of $\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{w-1}}$ such that $\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\boldsymbol{v}}_1=\dots=\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\boldsymbol{v}}_{\textcolor{\draftcolor}{w-1}}=0$.
 It means,  the input's dimension of the network is extended from $\textcolor{\draftcolor}{\ensuremath{d}}$ to $\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{w-1}$  and so the inner-layer matrix need also to be extended from $\textcolor{\draftcolor}{\ensuremath{\Theta}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times\textcolor{\draftcolor}{\ensuremath{d}}}$ to $[\textcolor{\draftcolor}{\ensuremath{\Theta}},\textcolor{\draftcolor}{A}]\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times(\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{w-1})}$.
 We also use the notation $\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}},\textcolor{\draftcolor}{A}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}$ to make an extended network that is also rescaled across the layers by a suitable $\textcolor{\draftcolor}{\boldsymbol{\alpha}}$.
 Note that the notation  $\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}},\textcolor{\draftcolor}{A}}$ is equivalnet with  $(\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{A}})_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}$, both means we rescale a matrix $\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{A}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times(\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{w-1})}$ with a vector $\textcolor{\draftcolor}{\boldsymbol{\alpha}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$ (see more details about rescaled networks in Section~\ref{AddNotations}).
 Using the above definitions, it is easy to see that  $\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}},\textcolor{\draftcolor}{A}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}=\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$, that means, the output of the
 extended and rescaled network is exactly the same as the original network (using the definition of $\textcolor{\draftcolor}{A}$ and rescaled weights).
In other words, we have a network that is first extended and then  rescaled while the output of the network is still the same as the original one.
We use the notation $\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}},\textcolor{\draftcolor}{A}}]~\deq~\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}[(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}},\textcolor{\draftcolor}{A}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}})^2]$ to compute the population risk in an extended and rescaled network.
We also define $\textcolor{\draftcolor}{p'}\deq\textcolor{\draftcolor}{\ensuremath{w}}+\textcolor{\draftcolor}{\ensuremath{w}}\cdot(\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{w-1})$ as the effective dimension of the extended network.

Now, let start the  proof by writing a second-order Taylor expansion of  $\objectiveFP{\textcolor{\draftcolor}{\parameteroutS_{\Sc}},\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}}$ (the risk in an extended and rescaled version of the target with $\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\parameteroutS_{\Sc}},\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}})\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p'}}$) around an extended and rescaled version of a reasonable stationary  $\textcolor{\draftcolor}{\ParamSta_{\ScTr}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}})\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p'}}$ with suitable  $\textcolor{\draftcolor}{\boldsymbol{\alpha}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$ and  $\textcolor{\draftcolor}{A},\textcolor{\draftcolor}{A}'\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times\textcolor{\draftcolor}{w-1}}$ (we see later how to assign suitable value for $\textcolor{\draftcolor}{\boldsymbol{\alpha}}$) to get

\begin{align*}
    \objectiveFP{\textcolor{\draftcolor}{\parameteroutS_{\Sc}},\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}}&=\objectiveFP{\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}}+\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveP}}[\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}]^\top(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})\\
    &~~~~~+\frac{1}{2}(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})^\top \HessianF{\textcolor{\draftcolor}{\StaOut_{\Sc}}+\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\parameteroutS_{\Sc}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}),\textcolor{\draftcolor}{\StaIn_{\ScTr}}+\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\StaIn_{\ScTr}})}(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})
\end{align*}
for some $\textcolor{\draftcolor}{t}\in(0,1)$~\citep[Proposition~1.1.13.a]{bertsekas2006convex}, where we use the notation $\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveP}}[\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}]\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p'}}$ and $\HessianF{\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}}\in \textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p'}\times\textcolor{\draftcolor}{p'}}$ to collect the first  and second order partial derivatives of  $\objectiveFP{\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}}$ with respect to the $\textcolor{\draftcolor}{\ParamSta_{\ScTr}}$, respectively (note that we have no assumption on $(\textcolor{\draftcolor}{\parameteroutS_{\Sc}},\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}})$ nor $(\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}})$ to have bounded norms).

Then, we employ the property of extended  and rescaled networks that is $\objectiveFP{\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}}=\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}$ and $\objectiveFP{\textcolor{\draftcolor}{\parameteroutS_{\Sc}},\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}}=\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}$, and use the shorthand notation
\begin{equation*}
 \textcolor{\draftcolor}{m}\deq(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})^\top \HessianF{\textcolor{\draftcolor}{\StaOut_{\Sc}}+\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\parameteroutS_{\Sc}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}),\textcolor{\draftcolor}{\StaIn_{\ScTr}}+\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\StaIn_{\ScTr}})}(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})
\end{equation*}
 to obtain
\begin{equation*}
    \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}=\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveP}}[\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}]^\top(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})+\frac{1}{2}\textcolor{\draftcolor}{m}\,.
\end{equation*}
Now, we are motivated to  show  that  $\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveP}}[\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}]^\top(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})=\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveP}}[\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}]^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})$.
To do so, we use 1.~our Lemma~\ref{HessianPOP} (for an extended and rescaled network),  2.~the property of   extended and rescaled networks, 3.~linearity of expectations, 4.~our assumption on $\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}$ that makes the first expectation zero, 5.~some rewriting, 6.~linearity of expectations, 7.~our assumption on $\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}$ (let recall that  $\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}=(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}^\top,\textcolor{\draftcolor}{\tilde{\datain}}^\top)^\top\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{w-1}}$ with $\textcolor{\draftcolor}{\tilde{\datain}}\sim\textcolor{\draftcolor}{\mathcal{N}(\boldsymbol{0},I_{\textcolor{\draftcolor}{w-1}\times\textcolor{\draftcolor}{w-1}})}$ that are also independent of $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$) that makes the second expectation zero, and 8.~some rewriting
to obtain that
\begingroup
\allowdisplaybreaks
\begin{align*}
    \frac{\partial}{\partial(\textcolor{\draftcolor}{\StaOut_{\Sc}})_{\textcolor{\draftcolor}{\ensuremath{j}}}}\objectiveFP{\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}}
&=-2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}^\top\textcolor{\draftcolor}{\StaIn_{\ScTr}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}\bigr)\bigl(\textcolor{\draftcolor}{\StaIn_{\ScTr}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr]\\
&=-2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\bigl(\textcolor{\draftcolor}{\StaIn_{\ScTr}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr]\\
&=-2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\textcolor{\draftcolor}{\ensuremath{y}}\bigl(\textcolor{\draftcolor}{\StaIn_{\ScTr}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr]+2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\bigl(\textcolor{\draftcolor}{\StaIn_{\ScTr}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr]\\
&=2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\bigl(\textcolor{\draftcolor}{\StaIn_{\ScTr}}\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr]\\
&=2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Biggl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{w}}-1}\bigl(\textcolor{\draftcolor}{\StaIn_{\ScTr}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr]\\
&=2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Biggl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\StaIn_{\ScTr}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr]+2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Biggl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=\textcolor{\draftcolor}{\ensuremath{d}}+1}^{\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{w}}-1}\bigl(\textcolor{\draftcolor}{\StaIn_{\ScTr}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\bar{\boldsymbol{\datainE}}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr]\\
&=2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Biggl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\StaIn_{\Sc}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr]\\
&=2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\bigl(\textcolor{\draftcolor}{\StaIn_{\Sc}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr]\,.
\end{align*}
\endgroup
Then, we 1.~imply our result above for all $\textcolor{\draftcolor}{\ensuremath{j}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$, 2.~use the definition of rescaled parameters and linearity of expectations to cancel $\textcolor{\draftcolor}{\boldsymbol{\alpha}}$'s, and 3.~use our results in Lemma~\ref{HessianPOP} to obtain
\begin{align*}
     \biggl( \frac{\partial}{\partial\textcolor{\draftcolor}{\StaOut_{\Sc}}}\objectiveFP{\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}}\biggr)^\top \bigl(\textcolor{\draftcolor}{\parameteroutS_{\Sc}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}\bigr)
     &=2\Bigl(\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\bigl(\textcolor{\draftcolor}{\StaIn_{\Sc}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\Bigr]\Bigr)^\top \bigl(\textcolor{\draftcolor}{\parameteroutS_{\Sc}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}\bigr)\\
     &=2\Bigl(\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\bigl(\textcolor{\draftcolor}{\widetilde\parameterin}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)\Bigr]\Bigr)^\top \bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}-\textcolor{\draftcolor}{\widetilde\parameterout}\bigr)\\
     &=\biggl(\frac{\partial}{\partial\textcolor{\draftcolor}{\widetilde\parameterout}}\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}\biggr)^\top (\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}-\textcolor{\draftcolor}{\widetilde\parameterout})\,.
\end{align*}
Implying a similar  argument as above for all partial derivatives, we  conclude that $\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveP}}[\textcolor{\draftcolor}{\StaOut_{\Sc}},\textcolor{\draftcolor}{\StaIn_{\ScTr}}]^\top(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})=\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveP}}[\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}]^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})$ (we omit the detailed proof). Tabulating this observation in the earlier display  we obtain

\begin{equation*}
    \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}=\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})+\frac{1}{2}\textcolor{\draftcolor}{m}\,.
\end{equation*}
 Rearranging  the display above we obtain
\begin{equation*}
   -\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})=\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+\frac{1}{2}\textcolor{\draftcolor}{m}\,.
\end{equation*}
Now, let recall the definition of stationary points in~\eqref{StaP} that implies
\begin{equation*}
   \textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})+\textcolor{\draftcolor}{\ensuremath{r}}\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\ge 0\,.
\end{equation*}

We 1.~rearrange above inequality and expand the bracket, 2.~use H\"older's inequality and the fact that $\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}^\top\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}=\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}$ (recall that $\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}\in\boldsymbol{\partial}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}$), and 3.~use $\normsup{\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}}\le 1$ to obtain
\begin{align*}
    -\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})&\le\textcolor{\draftcolor}{\ensuremath{r}}\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\ensuremath{r}}\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}^\top\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}\\
    &\le\textcolor{\draftcolor}{\ensuremath{r}}\normsupB{\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}\\
     &\le\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}\,,
\end{align*}
which by rearranging implies
\begin{align*}
    \textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}\ge 0\,.
\end{align*}
Display above reveals the positiveness of the terms in its left hand side and so we can obtain
\begin{equation*}
   -\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\le -\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})+ \textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}\,,
\end{equation*}
that is,
\begin{equation*}
   -\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\le \bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}\,.
\end{equation*}
Now, let use our display earlier (obtained by Taylor expansion)  rewriting the left hand side of the display above as
\begin{equation*}
  \objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+\frac{1}{2}\textcolor{\draftcolor}{m}\le \bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}\,.
\end{equation*}
 By some rearranging on the display above we obtain
 \begin{equation*}
  \objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}
   \le \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}+\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}-\frac{1}{2}\textcolor{\draftcolor}{m}\,.
\end{equation*}
For the right hand side of the inequality above we  1.~get an absolute value of the third term, 2.~add a zero-valued factor, 3.~use triangle inequality, and 4.~use our results in Lemma~\ref{EmpProcctwo}  to obtain
\begingroup
\allowdisplaybreaks
\begin{align*}
  \objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}
  &\le  \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}+\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\Bigr|-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}-\frac{1}{2}\textcolor{\draftcolor}{m}\\
  &=\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+2\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}+\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\Bigr|-\textcolor{\draftcolor}{\ensuremath{r}}\bigl(\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}\bigr)-\frac{1}{2}\textcolor{\draftcolor}{m}\\
    &\le \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+2\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}+\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\Bigr|-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}-\frac{1}{2}\textcolor{\draftcolor}{m}\\
   &\le  \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+2\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}+\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}} \normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\frac{\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}}{2\textcolor{\draftcolor}{\ensuremath{n}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}-\frac{1}{2}\textcolor{\draftcolor}{m}
\end{align*}
\endgroup
with probability at least   $1-1/2\textcolor{\draftcolor}{\ensuremath{n}}$.

The third and fifth terms in the last inequality above  can be canceled if we choose the tuning parameter large enough. Hence, we obtain
\begin{equation*}
  \objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}
    \le \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+2\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}+\frac{\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}}{2\textcolor{\draftcolor}{\ensuremath{n}}}-\frac{1}{2}\textcolor{\draftcolor}{m}
\end{equation*}
for $\textcolor{\draftcolor}{\ensuremath{r}}\ge \textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$.

The rest of the proof is analysing the behavior of $\textcolor{\draftcolor}{m}$.
Let rewrite $\textcolor{\draftcolor}{m}=\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}^{2}~\textcolor{\draftcolor}{m}'$
with
\begin{equation*}
    \textcolor{\draftcolor}{m}'~\deq~\frac{(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})^\top}{\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}} \HessianF{\textcolor{\draftcolor}{\StaOut_{\Sc}}+\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\parameteroutS_{\Sc}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}),\textcolor{\draftcolor}{\StaIn_{\ScTr}}+\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\StaIn_{\ScTr}})}\frac{(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})}{\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}}\,.
\end{equation*}

Now, we are motivated to employ our results in Proposition~\ref{PSDHessian}.
To do so,  we need   to make sure about invertibility of matrix $(\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}+\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\ensuremath{\Theta^*}}_{\textcolor{\draftcolor}{A}'}-\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}))(\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}+\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\ensuremath{\Theta^*}}_{\textcolor{\draftcolor}{A}'}-\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}))^\top$.
Using the definition of the extended networks, it is easy to see that $\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}$  and $\textcolor{\draftcolor}{\ensuremath{\Theta^*}}_{\textcolor{\draftcolor}{A}'}$ have full row rank.
Then, using Lemma~\ref{Inv_M},  we obtain that the line segment between two matrices $\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}$ and $\textcolor{\draftcolor}{\ensuremath{\Theta^*}}_{\textcolor{\draftcolor}{A}'}$ is not invertible at most in
finitely many $\textcolor{\draftcolor}{t}$.
It means, if we  shift  $\textcolor{\draftcolor}{t}$ by a tiny value $\varsigma\approx 0$  then, we can make sure that in the new point $\textcolor{\draftcolor}{t}'=\textcolor{\draftcolor}{t}-\varsigma$
 the corresponding matrix  is invertible, that is,
\begin{align*}
    \textcolor{\draftcolor}{m}'&~\deq~\frac{(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})^\top}{\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}} \HessianF{\textcolor{\draftcolor}{\StaOut_{\Sc}}+(\textcolor{\draftcolor}{t}-\varsigma+\varsigma)(\textcolor{\draftcolor}{\parameteroutS_{\Sc}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}),\textcolor{\draftcolor}{\StaIn_{\ScTr}}+(\textcolor{\draftcolor}{t}-\varsigma+\varsigma)(\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\StaIn_{\ScTr}})}\frac{(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})}{\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}}\\
    &\approx \frac{(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})^\top}{\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}} \HessianF{\textcolor{\draftcolor}{\StaOut_{\Sc}}+(\textcolor{\draftcolor}{t}-\varsigma)(\textcolor{\draftcolor}{\parameteroutS_{\Sc}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}),\textcolor{\draftcolor}{\StaIn_{\ScTr}}+(\textcolor{\draftcolor}{t}-\varsigma)(\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\StaIn_{\ScTr}})}\frac{(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})}{\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}}\\
    &= \frac{(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})^\top}{\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}} \HessianF{\textcolor{\draftcolor}{\StaOut_{\Sc}}+\textcolor{\draftcolor}{t}'(\textcolor{\draftcolor}{\parameteroutS_{\Sc}}-\textcolor{\draftcolor}{\StaOut_{\Sc}}),\textcolor{\draftcolor}{\StaIn_{\ScTr}}+\textcolor{\draftcolor}{t}'(\textcolor{\draftcolor}{\parameterinS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\StaIn_{\ScTr}})}\frac{(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})}{\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}}\,,
\end{align*}
where the second equation is reached by assuming $\varsigma$ is very close to zero and so we can ignore the remaining terms.
Then, we have $(\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}+\textcolor{\draftcolor}{t}'(\textcolor{\draftcolor}{\ensuremath{\Theta^*}}_{\textcolor{\draftcolor}{A}'}-\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}))(\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}+\textcolor{\draftcolor}{t}'(\textcolor{\draftcolor}{\ensuremath{\Theta^*}}_{\textcolor{\draftcolor}{A}'}-\textcolor{\draftcolor}{\widetilde\parameterin}_{\textcolor{\draftcolor}{A}}))^\top$ as an invertible matrix.

Implying
Proposition~\ref{PSDHessian} (with $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}=(\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}})/\normtwo{\textcolor{\draftcolor}{\ParamVS_{\Sc,\TransM'}}-\textcolor{\draftcolor}{\ParamSta_{\ScTr}}}$ and $\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{w-1}$ and $\textcolor{\draftcolor}{p'}$  as the dimension of the input and the effective dimension, respectively) we obtain that $\textcolor{\draftcolor}{m}'\in[0,\infty)$ for appropriate $\textcolor{\draftcolor}{\boldsymbol{\alpha}}$, that is, $\textcolor{\draftcolor}{\boldsymbol{\alpha}}$ with large enough $\textcolor{\draftcolor}{{c}}$).
The observation that $\textcolor{\draftcolor}{m}'\in[0,\infty)$ together with the definition of $\textcolor{\draftcolor}{m}$  implies that  $\textcolor{\draftcolor}{m}\in[0,\infty)$ as well.

Tabulating this observation to the display earlier together with our assumption on $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}$ ($\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}=\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}+\normoneM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}~\le~2\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}$) and  the fact that $1/2\textcolor{\draftcolor}{\ensuremath{n}}\le 4\sqrt{\log \textcolor{\draftcolor}{\ensuremath{n}}}$, we obtain for all $\textcolor{\draftcolor}{\ensuremath{r}}\ge \textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$ that
\begin{align*}
  \objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}
    &\le \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+2\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}+\frac{\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}}{2\textcolor{\draftcolor}{\ensuremath{n}}}-\frac{1}{2}\textcolor{\draftcolor}{m}\\
    &\le \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+2\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}+\frac{\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}}{2\textcolor{\draftcolor}{\ensuremath{n}}}\\
    &\le \objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}+5\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log \textcolor{\draftcolor}{\ensuremath{n}}}
    \end{align*}
with probability at least   $1-1/2\textcolor{\draftcolor}{\ensuremath{n}}$.

Second claim is a trivial consequence of the first claim by 1.~using $\textcolor{\draftcolor}{\ensuremath{r}}=\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$  and 2.~absorbing the constant 5 in $\textcolor{\draftcolor}{\nu}$ and simplifying to obtain
\begin{align*}
 \ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}]~&\leq~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+\textcolor{\draftcolor}{\nu}(\log{\textcolor{\draftcolor}{\ensuremath{n}}})^{3/2}\sqrt{\frac{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Bigl(5\sqrt{\log{\textcolor{\draftcolor}{\ensuremath{n}}}}\Bigr)\\
 & =~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+\textcolor{\draftcolor}{\nu}(\log{\textcolor{\draftcolor}{\ensuremath{n}}})^{2}\sqrt{\frac{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}{\textcolor{\draftcolor}{\ensuremath{n}}}}\,,
\end{align*}
with probability at least   $1-1/2\textcolor{\draftcolor}{\ensuremath{n}}$, which completes the proof.
\end{proof}
\subsubsection{Proof of Theorem~\ref{StaAppr}}
\begin{proof}
Main ingredients of the proof are the definition of $\textcolor{\draftcolor}{\tau}-$approximate stationary point and our Lemma~\ref{UBoundrisk}.

We start the proof using the definition of a $\textcolor{\draftcolor}{\tau}-$approximate stationary point in~\eqref{approx-sta} that implies
\begin{equation*}
    \objectiveF{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\dbtilde{\ParamV}}}~\le~\objectiveF{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\textcolor{\draftcolor}{\tau}\,.
\end{equation*}

We  add  zero-valued terms to the both sides of the inequality above to  obtain
\begin{equation*}
    \objectiveF{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}-\objectiveFP{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}+\objectiveFP{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\dbtilde{\ParamV}}}~\le~\objectiveF{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}-\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\textcolor{\draftcolor}{\tau}\,.
\end{equation*}
Then, we 1.~rearrange the terms, get an absolute value of the two terms, and use the properties of absolute values, 2.~get a supremum over the reasonable parameter space $\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}$ using our assumptions that $(\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}),(\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}~\deq~\{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}} : \normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}},\normoneM{\textcolor{\draftcolor}{\ensuremath{\Theta}}}\leq \sqrt{\log{\textcolor{\draftcolor}{\ensuremath{n}}}}\}$ (we use our assumption that the stationary is reasonable and our argument in the  paragraph above Theorem~\ref{StaAppr} to reach that $(\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}})$ is reasonable as well),  3.~simplify, and 4.~leave a negative term to obtain
\begingroup
\allowdisplaybreaks
\begin{align*}
    \objectiveFP{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}~&\le~\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+
    \Bigl|\objectiveF{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}-\objectiveFP{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}\Bigr|+
    \Bigl|\objectiveF{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}-\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}\Bigr|+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\dbtilde{\ParamV}}}+\textcolor{\draftcolor}{\tau}\\
    &\le~\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}}
    \bigl|\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\bigr|+\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}}
    \bigl|\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\bigr|\\
    &~~~~~~+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\dbtilde{\ParamV}}}+\textcolor{\draftcolor}{\tau}\\
     &=~\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+2\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}}
    \bigl|\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\bigr|+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\dbtilde{\ParamV}}}+\textcolor{\draftcolor}{\tau}\\
     &\le~\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+2\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}}
    \bigl|\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\bigr|+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\textcolor{\draftcolor}{\tau}\,.
\end{align*}
\endgroup

Then, we use 1.~our result above,  2.~Lemma~\ref{UBoundrisk}
bounding the second term with $\textcolor{\draftcolor}{t}=\textcolor{\draftcolor}{\nu}\sqrt{\log(32\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2)/\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}$ and $\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}=\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}$ (with probability at least $1-1/2\textcolor{\draftcolor}{\ensuremath{n}}$),
 3.~the definition of $\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}$ to replace $\allowbreak\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}}\normsupB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}}^2~\le~\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}}2\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}^2\normoneM{\textcolor{\draftcolor}{\ensuremath{\Theta}}}^2~\le~\allowbreak 2(\log\textcolor{\draftcolor}{\ensuremath{n}})^2\allowbreak~\eqd~\textcolor{\draftcolor}{\epsilon}'$, 4.~our Theorem~\ref{OracleInSta} upper bounding the first term (for $\textcolor{\draftcolor}{\ensuremath{r}}~\ge~\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$ with probability at least $1-1/2\textcolor{\draftcolor}{\ensuremath{n}}$),  5.~our assumption that stationary is reasonable, 6.~simplifying, 7.~an assumption that $\textcolor{\draftcolor}{\ensuremath{n}}~\ge~3$ (just for simplifying the terms), and 8.~the assumption that $\textcolor{\draftcolor}{\ensuremath{r}}~\ge~\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$ and the definition of $\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$ (note that for simplicity, we absorb all the  constants in $\textcolor{\draftcolor}{\nu}$) to obtain
\allowdisplaybreaks
\begingroup
\begin{align*}
    \objectiveFP{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}&\le~\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+2\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}_{\operatorname{res}}}}}
    \bigl|\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\bigr|+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\textcolor{\draftcolor}{\tau}\\
    &\le~\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+2\textcolor{\draftcolor}{\nu}\sqrt{\frac{\log(32\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2)}{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}\bigl(1+4\textcolor{\draftcolor}{\epsilon}'+4\sqrt{\textcolor{\draftcolor}{\epsilon}'}\bigr)+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\textcolor{\draftcolor}{\tau}\\
     &\le~\objectiveFP{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}+2\textcolor{\draftcolor}{\nu}\sqrt{\frac{\log(32\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2)}{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}\bigl(1+8(\log \textcolor{\draftcolor}{\ensuremath{n}})^2+8\log\textcolor{\draftcolor}{\ensuremath{n}}\bigr)+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\textcolor{\draftcolor}{\tau}\\
   &\le~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+5\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}+2\textcolor{\draftcolor}{\nu}\sqrt{\frac{\log(32\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2)}{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}\bigl(1+8(\log \textcolor{\draftcolor}{\ensuremath{n}})^2+8\log\textcolor{\draftcolor}{\ensuremath{n}}\bigr)+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\textcolor{\draftcolor}{\tau}\\
   &\le~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+5\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}+2\textcolor{\draftcolor}{\nu}\sqrt{\frac{\log(32\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2)}{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}\bigl(1+8(\log \textcolor{\draftcolor}{\ensuremath{n}})^2+8\log\textcolor{\draftcolor}{\ensuremath{n}}\bigr)+2\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log \textcolor{\draftcolor}{\ensuremath{n}}}+\textcolor{\draftcolor}{\tau}\\
    &=~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+7\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}+2\textcolor{\draftcolor}{\nu}\sqrt{\frac{\log(32\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2)}{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}\bigl(1+8(\log \textcolor{\draftcolor}{\ensuremath{n}})^2+8\log\textcolor{\draftcolor}{\ensuremath{n}}\bigr)+\textcolor{\draftcolor}{\tau}\\
    &\le~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+7\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}+34\textcolor{\draftcolor}{\nu}\sqrt{\frac{\log(32\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2)}{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}(\log \textcolor{\draftcolor}{\ensuremath{n}})^2+\textcolor{\draftcolor}{\tau}\\
     &\le~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+8\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}+\textcolor{\draftcolor}{\tau}\\
\end{align*}
\endgroup
with probability at least $1-(1+1)/2\textcolor{\draftcolor}{\ensuremath{n}}$, which is obtained by the fact that if $a~\le~ z_1+z_2$

\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}(a\le c_1+c_2)&\ge \textcolor{\draftcolor}{\mathbb{P}}(z_1+z_2\le c_1 + c_2)\\
    &=1-\textcolor{\draftcolor}{\mathbb{P}}(z_1+z_2 > c_1 + c_2)\\
    &\ge 1- \bigl(\textcolor{\draftcolor}{\mathbb{P}}(z_1 > c_1)+\textcolor{\draftcolor}{\mathbb{P}}(z_2 > c_2)\bigr)\,,
\end{align*}
where $a,z_1,z_2$ are random variables and $c_1,c_2$ are constants, as desired.

Second claim is a trivial consequence of the first claim by 1.~using $\textcolor{\draftcolor}{\ensuremath{r}}=\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$ and 2.~absorbing the constant 8 in $\textcolor{\draftcolor}{\nu}$ to obtain
\begin{align*}
 \ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}]~&~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+\textcolor{\draftcolor}{\nu}(\log{\textcolor{\draftcolor}{\ensuremath{n}}})^{3/2}\sqrt{\frac{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Bigl(8\sqrt{\log{\textcolor{\draftcolor}{\ensuremath{n}}}}\Bigr)+\textcolor{\draftcolor}{\tau}\\
 & =~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+\textcolor{\draftcolor}{\nu}(\log{\textcolor{\draftcolor}{\ensuremath{n}}})^{2}\sqrt{\frac{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}{\textcolor{\draftcolor}{\ensuremath{n}}}}+\textcolor{\draftcolor}{\tau}\,,
\end{align*}
with probability at least   $1-1/\textcolor{\draftcolor}{\ensuremath{n}}$, which completes the proof.

\end{proof}

\subsubsection{Proof of Proposition~\ref{PSDHessian}}
\begin{proof}

The proof is based on basic algebra and property of scaling weights across the layers in neural networks.

Let consider all the network parameters as a vector of length $\textcolor{\draftcolor}{p}$ (recall that $\textcolor{\draftcolor}{p}=\textcolor{\draftcolor}{\ensuremath{w}}+\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}}$). Then, we can tabulate the second order partial derivatives of $\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$ in a matrix called $\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p}\times\textcolor{\draftcolor}{p}}$ (for notational simplicity, we focus on $\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$ for the moment and then we move to $\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}$ at the end of the proof) of the form
\begin{equation*}
\allowdisplaybreaks
    \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}=\left[
\begin{array}{cc}
\textcolor{\draftcolor}{\ensuremath{A}}  & \textcolor{\draftcolor}{\ensuremath{B}} \\
 \textcolor{\draftcolor}{\ensuremath{C}} & \textcolor{\draftcolor}{\ensuremath{D}}
\end{array}\right]
\end{equation*}
with $\textcolor{\draftcolor}{\ensuremath{A}}\in\textcolor{\draftcolor}{\mathbb{R}}^{(\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}})\times(\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}})},~\textcolor{\draftcolor}{\ensuremath{B}}\in\textcolor{\draftcolor}{\mathbb{R}}^{(\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}})\times\textcolor{\draftcolor}{\ensuremath{w}}},~\textcolor{\draftcolor}{\ensuremath{C}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times(\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}})}$, and $\textcolor{\draftcolor}{\ensuremath{D}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times\textcolor{\draftcolor}{\ensuremath{w}}}$, where
\begingroup
\begin{align*}
\textcolor{\draftcolor}{\ensuremath{A}}_{(\textcolor{\draftcolor}{\ensuremath{j'}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k'}},(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}&\deq\frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\,,\\
\textcolor{\draftcolor}{\ensuremath{B}}_{(\textcolor{\draftcolor}{\ensuremath{j'}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k'}},\textcolor{\draftcolor}{\ensuremath{j}}}&\deq\frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k'}}}\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\,,\\
\textcolor{\draftcolor}{\ensuremath{C}}_{\textcolor{\draftcolor}{\ensuremath{j'}},(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}&\deq \frac{\partial^2}{\partial\textcolor{\draftcolor}{\parameteroutE_{j'}}\partial\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\,,\\
\textcolor{\draftcolor}{\ensuremath{D}}_{\textcolor{\draftcolor}{\ensuremath{j'}},\textcolor{\draftcolor}{\ensuremath{j}}}&\deq \frac{\partial^2}{\partial\textcolor{\draftcolor}{\parameteroutE_{j'}}\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}
\end{align*}
\endgroup
for $\textcolor{\draftcolor}{\ensuremath{j}},\textcolor{\draftcolor}{\ensuremath{j'}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$ and $\textcolor{\draftcolor}{\ensuremath{k}},\textcolor{\draftcolor}{\ensuremath{k'}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$.

 Applying the block-wise structure of $\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$, we are motivated to analyze the behavior of
\begin{equation*}
    \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}=(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{A}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}+(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{C}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}+(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{B}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}+(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{D}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\,.
\end{equation*}
Note  that $\textcolor{\draftcolor}{\ensuremath{C}}=\textcolor{\draftcolor}{\ensuremath{B}}^\top$ (see Lemma~\ref{HessianPOP}), so, we are left to analyze the behavior of
\begin{equation*}
    \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}=(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{A}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}+2(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{C}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}+(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{D}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}
\end{equation*}
for all $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p}}$ with $\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}}=1$.

We do the proof in steps:
We start by going through the three terms in the right hand side of display above  separately, to write them in a mathematically nice formulation (Steps~1:3). In Step~4, we sum up over the results calculated in Steps~1:3. Finally in Step~5, we use our result in Steps~1:4 to prove the main claims of the proposition.

\emph{Step~1:}\label{Ste1}
We  show that for $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}}}$ and $\textcolor{\draftcolor}{\ensuremath{A}}\in\textcolor{\draftcolor}{\mathbb{R}}^{(\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}})\times(\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}})}$,
\begin{equation*}
     (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{A}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}
    =2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2\,,
\end{equation*}
where we denote $(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}:=\bigl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{\textcolor{\draftcolor}{\ensuremath{k}}},(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}},\dots,(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{w}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\bigr)^\top\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$ (as a sub-vector of $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}$) for each $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$.

 We start by writing  matrix product in the form of sums and fill the entries of matrix $\textcolor{\draftcolor}{\ensuremath{A}}$ with the corresponding values from the definition to get
\begin{align*}
    (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{A}}&\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\\
    &=\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k'}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j'}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k'}}}\frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\biggr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\,.
\end{align*}
By Lemma~\ref{HessianPOP} we have
\begin{equation*}
    \frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}=2\textcolor{\draftcolor}{\parameteroutE_{j'}}\textcolor{\draftcolor}{\parameteroutE_j}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k'}}}\bigr]\,,
\end{equation*}
which  using our assumption on  $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$ (identity covariance matrix) implies
\begin{equation*}
    \frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}=2\textcolor{\draftcolor}{\parameteroutE_{j'}}\textcolor{\draftcolor}{\parameteroutE_j}
\end{equation*}

for $\textcolor{\draftcolor}{\ensuremath{k}}=\textcolor{\draftcolor}{\ensuremath{k'}}$ and zero otherwise (for $\textcolor{\draftcolor}{\ensuremath{k}}\ne\textcolor{\draftcolor}{\ensuremath{k'}}$).
We use 1.~our display earlier, 2.~the result above, 3.~the linearity of sums, 4.~some rewriting (using multinomial theorem), and 5.implying our notation $(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}$ for writing the sum in the form of  product  to obtain
\begingroup
\allowdisplaybreaks
\begin{align*}
    (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{A}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}
    &=\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k'}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j'}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k'}}}\frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\biggr)\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j'}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\textcolor{\draftcolor}{\parameteroutE_{j'}} \textcolor{\draftcolor}{\parameteroutE_j}\Bigr)\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j'}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\textcolor{\draftcolor}{\parameteroutE_{j'}}\textcolor{\draftcolor}{\parameteroutE_j} \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\bigr)\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\textcolor{\draftcolor}{\parameteroutE_j}\Biggr)^2\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2\,.
\end{align*}
\endgroup

\emph{Step~2:}\label{stp2}
We  prove that for $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$ and $\textcolor{\draftcolor}{\ensuremath{D}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times\textcolor{\draftcolor}{\ensuremath{w}}}$,
\begin{equation*}
     (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{D}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}
 =2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\Bigr)^2,
\end{equation*}
where $\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}}$ denotes the $\textcolor{\draftcolor}{\ensuremath{k}}$-th column of $\textcolor{\draftcolor}{\ensuremath{\Theta}}$.

For each $\textcolor{\draftcolor}{\ensuremath{j}},\textcolor{\draftcolor}{\ensuremath{j'}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$, we use 1.~the result of Lemma~\ref{HessianPOP}, 2.~the definition of covariance, 3.~the fact that $\text{Cov}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})=\textcolor{\draftcolor}{\ensuremath{\Theta}}\text{Cov}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})\textcolor{\draftcolor}{\ensuremath{\Theta}}^\top$, 4.~the assumption on $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$ (identity covariance), and 5.~rewriting  to obtain
\begin{align*}
 \frac{\partial^2}{\partial\textcolor{\draftcolor}{\parameteroutE_{j'}}\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}
 &=2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\bigr]\\
 &=2\bigl(\text{Cov}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})\bigr)_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{j}}}\\
 &=2\bigl(\textcolor{\draftcolor}{\ensuremath{\Theta}}\text{Cov}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})\textcolor{\draftcolor}{\ensuremath{\Theta}}^\top\bigr)_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{j}}}\\
 &=2\bigl(\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top\bigr)_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{j}}}\\
 &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k}}}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}\,.
\end{align*}
We use 1.~the definition of sub-matrix $\textcolor{\draftcolor}{\ensuremath{D}}$ to write the matrix product in the form of a sum, 2.~tabulating above result and using the linearity of sums, 3.~some rewriting (using the multinomial theorem),  and 4.~writing the sum in the form of  product to obtain
\begingroup
\allowdisplaybreaks
\begin{align*}
  (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{D}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}
  &=\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}} \biggl( (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j'}}} \frac{\partial^2}{\partial\textcolor{\draftcolor}{\parameteroutE_{j'}}\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\biggr)\\
  &=\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}} 2(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k}}}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\\
  &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\bigr)\Biggr)^2\\
  &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\Bigr)^2\,.
\end{align*}
\endgroup

\emph{Step~3:} \label{stp3}
We  show that for
 $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$, $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\in R^{\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}}}$, and $\textcolor{\draftcolor}{\ensuremath{C}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times(\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}})}$,
\begin{align*}
 (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{C}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}&=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\Bigr)\\
    &~~~~~+2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\biggl(\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\Bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr).
\end{align*}

Expanding $(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{C}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}$ yields

\begin{equation*}
    (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{C}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}=\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}\frac{\partial^2}{\partial\textcolor{\draftcolor}{\gamma}_{\textcolor{\draftcolor}{\ensuremath{j'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\biggr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr).
\end{equation*}

Now, we need to consider two different cases:

\emph{Case~1:} ($\textcolor{\draftcolor}{\ensuremath{j}}\ne\textcolor{\draftcolor}{\ensuremath{j'}}$)

We use 1.~the result of Lemma~\ref{HessianPOP}, 2.~writing matrix product in the form of a sum, 3.~linearity of sums and expectations, and 4.~our assumption on $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$  to get for each $\textcolor{\draftcolor}{\ensuremath{j}},\textcolor{\draftcolor}{\ensuremath{j'}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$ and $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$ with $\textcolor{\draftcolor}{\ensuremath{j}}\ne\textcolor{\draftcolor}{\ensuremath{j'}}$ that

\begin{align*}
    \frac{\partial^2}{\partial\textcolor{\draftcolor}{\gamma}_{\textcolor{\draftcolor}{\ensuremath{j'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}
    &=2\textcolor{\draftcolor}{\parameteroutE_j}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}\bigr]\\
    &=2\textcolor{\draftcolor}{\parameteroutE_j}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Biggl[(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k'}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k'}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k'}}}\bigr)\Biggr]\\
     &=2\textcolor{\draftcolor}{\parameteroutE_j}\sum_{\textcolor{\draftcolor}{\ensuremath{k'}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k'}}}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k'}}}\bigr]\Bigr)\\
    &=2\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k}}}\,.
\end{align*}

\emph{Case~2:} ($\textcolor{\draftcolor}{\ensuremath{j}}=\textcolor{\draftcolor}{\ensuremath{j'}}$)

We  use 1.~the result of Lemma~\ref{HessianPOP}, 2.~linearity of expectations, 3.~linearity of expectations and our assumption on $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$ (same argument as above), 4.~linearity of expectations, 5.~linearity of expectations and our  assumption on $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$, 6.~adding a zero-valued term, and 7.~again linearity of expectations, our assumption on $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$, and some rearranging  to obtain

\begingroup
\allowdisplaybreaks
\begin{align*}
    \frac{\partial^2}{\partial\textcolor{\draftcolor}{\gamma}_{\textcolor{\draftcolor}{\ensuremath{j}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}
  &=2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\textcolor{\draftcolor}{\parameteroutE_j}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{j}}}-\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\\
  &=2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\textcolor{\draftcolor}{\parameteroutE_j}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\bigr]+2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\textcolor{\draftcolor}{\ensuremath{y}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\\
  &=2\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}+2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\textcolor{\draftcolor}{\ensuremath{y}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\\
  &=2\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}+2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]-2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\textcolor{\draftcolor}{\ensuremath{y}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\\
   &=2\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}+2\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\textcolor{\draftcolor}{\ensuremath{y}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\\
   &=2\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}+2\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}+\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\\
   &=2\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}+2\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-2\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\,.
\end{align*}
\endgroup
Now, we 1.~use our  earlier expansion, 2.~separate the innermost sum in two cases, 3.~use the result  above (Case~1 and Case~2), 4.~do some rearranging, 5.~use linearity of sums and some rewriting, and 6.~write sums in the form of vector product and rearranging  to obtain

\begingroup
\allowdisplaybreaks
\begin{align*}
    (\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}&)^\top\textcolor{\draftcolor}{\ensuremath{C}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\\
    &=\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}\frac{\partial^2}{\partial\textcolor{\draftcolor}{\gamma}_{\textcolor{\draftcolor}{\ensuremath{j'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\biggr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\\
    &=\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1,\textcolor{\draftcolor}{\ensuremath{j'}}\ne\textcolor{\draftcolor}{\ensuremath{j}}}^{\textcolor{\draftcolor}{\ensuremath{w}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}\frac{\partial^2}{\partial\textcolor{\draftcolor}{\gamma}_{\textcolor{\draftcolor}{\ensuremath{j'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\biggr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\\
    &~~~~~+\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\frac{\partial^2}{\partial\textcolor{\draftcolor}{\gamma}_{\textcolor{\draftcolor}{\ensuremath{j}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1,\textcolor{\draftcolor}{\ensuremath{j'}}\ne\textcolor{\draftcolor}{\ensuremath{j}}}^{\textcolor{\draftcolor}{\ensuremath{w}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)\\
    &~~~~~+2\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigl(\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}+\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\Bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\biggr)\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}\textcolor{\draftcolor}{\parameteroutE_j}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)\\
    &~~~~~+2\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\Bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\biggr)\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j'}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j'}}\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\textcolor{\draftcolor}{\parameteroutE_j}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\\
    &~~~~~+2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\Bigr)\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})_{\textcolor{\draftcolor}{\ensuremath{j}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})_{(\textcolor{\draftcolor}{\ensuremath{j}}-1)\textcolor{\draftcolor}{\ensuremath{d}}+\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr)\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})\Bigr)\\
    &~~~~~+2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\Bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\,.
\end{align*}
\endgroup

\emph{Step~4:}\label{stp4}
We  prove that for any $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}=[(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top,(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top]^\top\in \textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p}}$ and $(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}$, it holds that
\begin{align*}
 \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}
 &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})+\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2
    +4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\\
     &~~~~~-4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr](\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\,.
\end{align*}

We use 1.~the block-wise structure of the Hessian matrix and some rearranging, 2.~our results in Steps~1:3, and 3.~multinomial theorem to obtain

\begingroup
\allowdisplaybreaks
\begin{align*}
    \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}
    &=(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{A}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}+(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{D}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}+2(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{C}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2
    +2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\Bigr)^2
    +4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\Bigr)\\
    &~~~~~+4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\biggl(\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr]\Bigr)(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr)\\
    &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}+\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2
    +4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\\
     &~~~~~-4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr](\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\,.
\end{align*}
\endgroup

\noindent\emph{Step 5: }Now, we employ our result in Steps~1:4 to prove the main claims of the proposition.

\emph{Claim~1:}\label{Claimtwo} ($\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\ne\boldsymbol{0}$  and $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}=\boldsymbol{0}$)

We use 1.~the block-wise structure of the Hessian, 2.~the assumption that  $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}=\boldsymbol{0}$, 3.~our result in Step~1, and 4.~the fact that sum of non-negative terms is also non-negative  to obtain
\begin{align*}
 \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}&=(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{A}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}+2(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{C}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}+(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{D}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\\
 &=(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top\textcolor{\draftcolor}{\ensuremath{A}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\\
 &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2\\
 &\ge 0\,.
\end{align*}
 Above display can also reveal that for all $\textcolor{\draftcolor}{\boldsymbol{\alpha}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}\setminus\{\boldsymbol{0}\}$   (moving to a scaled version of the parameters)
 \begin{equation*}
 \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}
 =2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2~\ge~0\,,
\end{equation*}
 as desired.

\emph{Claim~2:} ($\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}=\boldsymbol{0}$  and $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\ne\boldsymbol{0}$)

The proof is similar with~\emph{Claim~1} and so we omit the proof.

 \emph{Claim~3:} ($\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\ne \boldsymbol{0}$ and $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\ne\boldsymbol{0}$)

We use our results in Step~4 together with getting an absolute value of the two last terms to  obtain
\begingroup
\allowdisplaybreaks
\begin{align*}
 \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}
 &=2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}+\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2
    +4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\\
     &~~~~~-4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr](\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\\
     &\ge~2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}+\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2
    -4\Biggl|\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr|\\
     &~~~~~-4\Biggl|\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr](\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr|\,.
\end{align*}
\endgroup

First let us concentrate on the second term of display above and 1.~use the triangle inequality and properties of absolute values, 2.~use H\"{o}lder inequality, 3.~get a factor $\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}$ out of the summation, 4.~use Cauchy–Schwarz inequality, and 5. some rewriting to obtain
\begingroup
\allowdisplaybreaks
\begin{align*}
    4\Biggl|\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr|
    &\le 4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl|\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr|\Bigl|(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr|\\
    &\le 4\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl|\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr|\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}}\\
    &= 4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl|\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr|\normtwoB{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}}\\
    &\le 4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\sqrt{\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl|\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\bigr)_{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr|^2}\sqrt{\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\normtwoB{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}}^2}\\
    &= 4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\,.
\end{align*}
\endgroup

Then, we use 1.~our assumption that~$\textcolor{\draftcolor}{\ensuremath{y}}=\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}+\textcolor{\draftcolor}{u}$, 2.~independence of $\textcolor{\draftcolor}{u}$ and $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$, and 3.~our assumption that~$\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}]=\boldsymbol{0}$ (also we have $\mathbb{E}[\textcolor{\draftcolor}{u}]=0$) to obtain
\begin{align*}
    4\Biggl|\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr](\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr|
    &=4\Biggl|\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\textcolor{\draftcolor}{u}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr](\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr|\\
    &=4\Biggl|\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[\textcolor{\draftcolor}{u}\bigr]\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\bigl[(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr](\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Biggr|\\
    &=0\,.
\end{align*}
Tabulating  two observations above in the previous display we obtain
\begin{align*}
    \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}&\ge 2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}+\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)^2
    -4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\,.
\end{align*}

Now, let define for each $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$ that $A_k\deq(\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}$, $B_k\deq\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}$, and
 using the fact $(A_k+B_k)^2\ge \frac12 (A_k)^2-(B_k)^2$ to obtain
\begin{align*}
     \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top &\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}\\
     &\ge 2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}(A_k+B_k)^2 -4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\\
     &\ge \sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}(A_k)^2 - 2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}(B_k)^2 -4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\,.
\end{align*}
Now, we analyze the first two terms in the right hand side of the last inequality above.
We use 1.~the definition of $A_k$, 2.~some rewritings, 3.~the linearity of sums, 4.~the definition of matrix product, 5.~property of eigenvalues ($\textcolor{\draftcolor}{{e}_{\operatorname{min}}}[\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top]$ denotes the smallest eigenvalue of $\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top$), and 6.~the norm definition to obtain
\begingroup
\allowdisplaybreaks
\begin{align*}
    \sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}(A_k)^2&=\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\Bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\Bigr)^2\\
    &=\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\\
    &=(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}_{.,\textcolor{\draftcolor}{\ensuremath{k}}})^\top\Biggr)\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\\
    &=(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\\
    &\ge\textcolor{\draftcolor}{{e}_{\operatorname{min}}}\bigl[\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top\bigr](\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\\
    &=\textcolor{\draftcolor}{{e}_{\operatorname{min}}}\bigl[\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top\bigr]\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}^2\,.
\end{align*}
\endgroup

Also, using 1.~the definition of $B_k$, 2.~the Cauchy–Schwarz inequality, 3.~the linearity of sums, and 4.~the definition of norms we obtain
\begin{align*}
    2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}(B_k)^2= 2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr)^2
    \le 2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}^2\normtwoB{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}}^2=2\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}^2\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\normtwoB{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^{\textcolor{\draftcolor}{\ensuremath{k}}}}^2=2\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}^2\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}^2\,.
\end{align*}
Collecting two  displays above together with the earlier one we obtain
\begin{equation*}
    \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top \HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}
     \ge\textcolor{\draftcolor}{{e}_{\operatorname{min}}}\bigl[\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top\bigr]\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}^2-2\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}^2\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}^2 -4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\,.
\end{equation*}

Now, it is time to concentrate on the Hessian behavior of   $\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}$ (and not $\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$).
We  use the known fact in neural networks that weights can be rescaled across the layers once activations are nonnegative-homogeneous. It says for a neural network parameterized by $(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})$, there is another network with the same objective value such that the covariates of $\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}$ are multiplied  by the covariates of $\boldsymbol{\textcolor{\draftcolor}{\alpha}}$ and  the covariates in each column of $\textcolor{\draftcolor}{\ensuremath{\Theta}}$ are divided by the covariates of $\boldsymbol{\textcolor{\draftcolor}{\alpha}}$.
We use this fact with $\textcolor{\draftcolor}{\alphaN_{j}}=1/\textcolor{\draftcolor}{{c}}$ for all $\textcolor{\draftcolor}{\ensuremath{j}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$, which $\textcolor{\draftcolor}{{c}}\in(1,\infty)$, together with the above result to analyse the behavior of Hessian in $(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}})$ and get
\begin{align*}
 \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}} &\ge \textcolor{\draftcolor}{{e}_{\operatorname{min}}}\bigl[\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}{\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}^\top\bigr]\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}^2-2\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}^2\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}^2 -4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}-{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}}}\\
 &=\textcolor{\draftcolor}{{c}}^2\textcolor{\draftcolor}{{e}_{\operatorname{min}}}\bigl[\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top\bigr]\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}^2-\frac{2}{\textcolor{\draftcolor}{{c}}^2}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}^2\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}^2 -4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\,,
  \end{align*}
where for the last line we use factorizing and the definition of scaled parameters.
Using above display,  we can guarantee positive semidefinite Hessian once  $\textcolor{\draftcolor}{{c}}$ is selected large enough because, the first term can dominate the other two terms.
So, we use $\textcolor{\draftcolor}{{c}}\in[1,\infty)$ and our assumption on $\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top$ to obtain that for
\begin{equation*}
    \textcolor{\draftcolor}{{c}}^2 \ge \frac{2\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}^2\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}^2 +4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}}{\textcolor{\draftcolor}{{e}_{\operatorname{min}}}\bigl[\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top\bigr]\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}^2}\,,
\end{equation*}
we can guarantee positive semidifinite Hessian, as desired.
\end{proof}

\subsubsection{Proof of Lemma~\ref{EmpProcctwo}}
\begin{proof}
The  proof idea is inspired by~\citet[Lemma~14]{elsener2018sharp} and main ingredients  are our Lemma~\ref{Empproc} and union bounds.

Let define
$\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\deq2\textcolor{\draftcolor}{t}$ for $\textcolor{\draftcolor}{t}\in(0,\infty)$, $\textcolor{\draftcolor}{s_{\paramspace}}\deq(\textcolor{\draftcolor}{\eta}+\max\{\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}, \normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\})(1+\textcolor{\draftcolor}{\epsilon})$, which is basically defined by  parameters $\textcolor{\draftcolor}{\epsilon}$ and $\textcolor{\draftcolor}{\eta}$ of $\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}$ (recall that $\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}=\{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p}}: \normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}\le \textcolor{\draftcolor}{\eta} \text{~~and~~} \normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\leq \textcolor{\draftcolor}{\epsilon} \}$), and $\textcolor{\draftcolor}{Z(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}$ as a function of two vectors $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}$ and $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}$ (with $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})$) defined as
\begin{equation*}
    \textcolor{\draftcolor}{Z(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}\deq\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}})\Bigr|\,.
\end{equation*}
Using Lemma~\ref{Empproc} and notations above and with assuming $\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}$ (specific values of $\textcolor{\draftcolor}{\epsilon}$ and $\textcolor{\draftcolor}{\eta}$ be assigned at the end of the proof) we obtain for each $\textcolor{\draftcolor}{t}\in(0,\infty)$ that
\begin{equation*}
    \textcolor{\draftcolor}{\mathbb{P}}\Bigl(\textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}~\ge~\textcolor{\draftcolor}{\eta}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}\Bigr)
    ~\le~\textcolor{\draftcolor}{\mathbb{P}}\biggl(\sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}} \textcolor{\draftcolor}{Z(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}~\ge~ \textcolor{\draftcolor}{\eta}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}\biggr)
    ~\le~ 4\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\littlespace e^{-\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}\min\{{\textcolor{\draftcolor}{t}^2}/{\textcolor{\draftcolor}{\nu}^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}}\}}
\end{equation*}
with $\textcolor{\draftcolor}{\nu},\textcolor{\draftcolor}{\kappa}\in(0,\infty)$ constants depending only on the distributions of the inputs and noises.

We assume without lose of generality that $1/\textcolor{\draftcolor}{\ensuremath{n}}\le \textcolor{\draftcolor}{\eta}$ and continue the proof in two different cases:

\emph{Case~1:}~$(\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~ 1/\textcolor{\draftcolor}{\ensuremath{n}})$

In this case, we  use 1.~the fact that $\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}\geq 0$, 2.~our assumption that $1/\textcolor{\draftcolor}{\ensuremath{n}}~\le~\textcolor{\draftcolor}{\eta}$ and the definition of $\textcolor{\draftcolor}{s_{\paramspace}}$, and  3.~our assumption that $\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~1/\textcolor{\draftcolor}{\ensuremath{n}}$, which implies that $\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}\in\mathcal{C}_{1/n,\textcolor{\draftcolor}{\epsilon}}$ and our argument above
to obtain for each $\textcolor{\draftcolor}{t}\in(0,\infty)$ that
\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}\biggl(\textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}\ge 2\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}+\frac{\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{s_{\paramspace}}\biggr)
    &~\le~\textcolor{\draftcolor}{\mathbb{P}}\biggl(\textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}\ge \frac{\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{s_{\paramspace}}\biggr)\\
    &~\le~\textcolor{\draftcolor}{\mathbb{P}}\biggl(\textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}\ge \frac{\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}}{\textcolor{\draftcolor}{\ensuremath{n}}}s_{\mathcal{C}_{1/n,\textcolor{\draftcolor}{\epsilon}}}\biggr)\\
  &~\le~4\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\littlespace e^{-\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}\min\{{\textcolor{\draftcolor}{t}^2}/{\textcolor{\draftcolor}{\nu}^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}}\}}\,.
\end{align*}

\emph{Case~2:}~$( 1/\textcolor{\draftcolor}{\ensuremath{n}}~<~\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~\textcolor{\draftcolor}{\eta})$

In this case, we use 1.~the fact that for mutually exclusive events $H_1,\dots,H_n$: $\textcolor{\draftcolor}{\mathbb{P}}(\cup_{i=1}^{n}H_i)=\sum_{i=1}^{n}\textcolor{\draftcolor}{\mathbb{P}}(H_i)$,
 2.~lower bound of $\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}$, 3.~the fact that $\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}/\textcolor{\draftcolor}{\ensuremath{n}}~\ge~0$ and removing the lower bound, 4.~the fact that $2^{i+1}/\textcolor{\draftcolor}{\ensuremath{n}}~\le ~\textcolor{\draftcolor}{\eta}$, and 5.~the fact that $\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}\in s_{\mathcal{C}_{2^{i+1}/\textcolor{\draftcolor}{\ensuremath{n}},\textcolor{\draftcolor}{\epsilon}}}$ and our earlier argument to obtain for each $\textcolor{\draftcolor}{t}\in(0,\infty)$ that
 \begingroup
 \allowdisplaybreaks
\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}&\biggl(\textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}~\ge~ 2\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}+\frac{\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{s_{\paramspace}}~~\text{for}~~\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}~<~\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~\textcolor{\draftcolor}{\eta}\biggr)\\
    &~=~\sum_{i=0}^{\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta}  )}\rceil-1}\textcolor{\draftcolor}{\mathbb{P}}\biggl(\textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}~\ge~ 2\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}+\frac{\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{s_{\paramspace}}~~\text{for}~~\frac{2^i}{\textcolor{\draftcolor}{\ensuremath{n}}}~<~\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~ \frac{2^{i+1}}{\textcolor{\draftcolor}{\ensuremath{n}}}\biggr)\\
     &~\le~ \sum_{i=0}^{\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta}  )}\rceil-1}\textcolor{\draftcolor}{\mathbb{P}}\biggl( \textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}~\ge~ \frac{2^{i+1}}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}+\frac{\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{s_{\paramspace}}~~\text{for}~~\frac{2^i}{\textcolor{\draftcolor}{\ensuremath{n}}}~<~ \normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~\frac{2^{i+1}}{\textcolor{\draftcolor}{\ensuremath{n}}}\biggr)\\
      &~\le~ \sum_{i=0}^{\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta}  )}\rceil-1}\textcolor{\draftcolor}{\mathbb{P}}\biggl( \textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}~\ge~ \frac{2^{i+1}}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}\textcolor{\draftcolor}{s_{\paramspace}}~~\text{for}~~\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~ \frac{2^{i+1}}{\textcolor{\draftcolor}{\ensuremath{n}}}\biggr)\\
      &~\le~ \sum_{i=0}^{\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta}  )}\rceil-1}\textcolor{\draftcolor}{\mathbb{P}}\biggl( \textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}~\ge~ \frac{2^{i+1}}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})} s_{\mathcal{C}_{2^{i+1}/\textcolor{\draftcolor}{\ensuremath{n}},\textcolor{\draftcolor}{\epsilon}}}~~\text{for}~~\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~ \frac{2^{i+1}}{\textcolor{\draftcolor}{\ensuremath{n}}}\biggr)\\
      &~\le~ 4\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta} )}\rceil \textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\littlespace e^{-\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}\min\{{\textcolor{\draftcolor}{t}^2}/{\textcolor{\draftcolor}{\nu}^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}}\}}\,.
\end{align*}
\endgroup
We collect all pieces of the proof (\emph{Case~1} and \emph{Case~2}),  set $\textcolor{\draftcolor}{t}=\textcolor{\draftcolor}{\nu}\sqrt{\log{(8\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta} )}\rceil)}/(\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}})}$ (we use the notation $\log$ as natural logarithm), and use the union bounds to obtain (we also need to assume $\textcolor{\draftcolor}{\ensuremath{n}}$ is large enough to get rid of the min operator)
\begin{align*}
     \textcolor{\draftcolor}{\mathbb{P}}\bBigg@{3.5}( \textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}&~\ge~ 2\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}\tilde{\textcolor{\draftcolor}{\ensuremath{r}}}\Bigl(\textcolor{\draftcolor}{\nu}\sqrt{\log{\bigl(8\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta} )}\rceil\bigr)}/(\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}})}\Bigr)\textcolor{\draftcolor}{s_{\paramspace}}\\
     &~~~~~~~~~~~+\frac{\tilde{\textcolor{\draftcolor}{\ensuremath{r}}}\Bigl(\textcolor{\draftcolor}{\nu}\sqrt{\log{\bigl(8\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta} )}\rceil\bigr)}/(\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}})}\Bigr)}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{s_{\paramspace}}\bBigg@{3.5})\\
     &~\le~ 4\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta} )}\rceil \textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\littlespace e^{-\log(8\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta} )}\rceil)}\\
     &~=~\frac{1}{2\textcolor{\draftcolor}{\ensuremath{n}}}\,.
\end{align*}
Now, we use the results above and the definitions of $\textcolor{\draftcolor}{Z(\ParamSta,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}})}$ and $\textcolor{\draftcolor}{\tilde{\tuning}(\textcolor{\draftcolor}{t})}$  to obtain

\begin{align*}
     \textcolor{\draftcolor}{\mathbb{P}}\bBigg@{3.5}(\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\Bigr|&~\ge~ 4\textcolor{\draftcolor}{\nu}\textcolor{\draftcolor}{s_{\paramspace}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}\sqrt{\frac{\log\bigl(8\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta} )}\rceil\bigr)}{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}\\
     &~~~~~~~+2\textcolor{\draftcolor}{\nu}\textcolor{\draftcolor}{s_{\paramspace}}\sqrt{\frac{\log\bigl(8\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\lceil\log_{2}{(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\eta} )}\rceil\bigr)}{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}^3}}\bBigg@{3.5})\\
     &~\le~ \frac{1}{2\textcolor{\draftcolor}{\ensuremath{n}}}\,.
\end{align*}
Then, we use our assumption that the  stationary point $(\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin})$ is reasonable to obtain: $\normone{\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}~\leq~ \normone{\textcolor{\draftcolor}{\widetilde\parameterout}^\top\textcolor{\draftcolor}{\widetilde\parameterin}}+\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}~\le~ \normone{\textcolor{\draftcolor}{\widetilde\parameterout}}\normsupM{\textcolor{\draftcolor}{\widetilde\parameterin}}+\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}\normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}~\le~2\log\textcolor{\draftcolor}{\ensuremath{n}}$ (using triangle inequality, H\"{o}lder's inequality, and our assumption on reasonable target and stationary) and $\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~ \normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~=~\normone{\textcolor{\draftcolor}{\widetilde\parameterout}}+\normoneM{\textcolor{\draftcolor}{\widetilde\parameterin}}+\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}+\normoneM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}~\le~   4\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}$ (using triangle inequality, our definition of norm, and our assumption on reasonable target and stationary), which means we can assign $\textcolor{\draftcolor}{\epsilon}= 2\log\textcolor{\draftcolor}{\ensuremath{n}}$ and $\textcolor{\draftcolor}{\eta}=4\sqrt{\log \textcolor{\draftcolor}{\ensuremath{n}}}$ (for $\textcolor{\draftcolor}{\ensuremath{n}}~\ge~2$ we can make sure that $1/\textcolor{\draftcolor}{\ensuremath{n}}~\le~ \textcolor{\draftcolor}{\eta}$ is satisfied).

Now, we plug in the values of $\textcolor{\draftcolor}{\epsilon}= 2\log\textcolor{\draftcolor}{\ensuremath{n}}$,   $\textcolor{\draftcolor}{\eta}=4\sqrt{\log \textcolor{\draftcolor}{\ensuremath{n}}}$, and $\textcolor{\draftcolor}{s_{\paramspace}}=(\textcolor{\draftcolor}{\eta}+\max\{\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}, \normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\})(1+\textcolor{\draftcolor}{\epsilon})~\le~(5\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}})(1+2\log\textcolor{\draftcolor}{\ensuremath{n}})~\le~15(\log \textcolor{\draftcolor}{\ensuremath{n}})^{3/2}$ (for $\textcolor{\draftcolor}{\ensuremath{n}}~\ge~2$) to conclude that
\begin{align*}
     \textcolor{\draftcolor}{\mathbb{P}}\biggl(\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}&-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\Bigr|\\
     &~\ge~\frac{60}{\sqrt{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}\textcolor{\draftcolor}{\nu}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}(\log\textcolor{\draftcolor}{\ensuremath{n}})^{3/2}\sqrt{{\log\bigl(8\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\lceil\log_{2}{(4\textcolor{\draftcolor}{\ensuremath{n}}\sqrt{\log \textcolor{\draftcolor}{\ensuremath{n}}} )}\rceil\bigr)}}\\
     &~~~~~~~~+\frac{30}{\sqrt{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}^3}}\textcolor{\draftcolor}{\nu}(\log\textcolor{\draftcolor}{\ensuremath{n}})^{3/2}\sqrt{{\log\bigl(8\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p}\lceil\log_{2}{(4\textcolor{\draftcolor}{\ensuremath{n}}\sqrt{\log \textcolor{\draftcolor}{\ensuremath{n}}} )}\rceil\bigr)}}\biggr)\\
     &~\le~\frac{1}{2\textcolor{\draftcolor}{\ensuremath{n}}}\,.
\end{align*}
Then, we use the fact that $\textcolor{\draftcolor}{\ensuremath{d}}\le \textcolor{\draftcolor}{p}$ and simplifying display above to obtain
\begin{align*}
     \textcolor{\draftcolor}{\mathbb{P}}\biggl(\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}&-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\Bigr|\\
     &~\ge~\frac{180}{\sqrt{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}}}\textcolor{\draftcolor}{\nu}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}(\log\textcolor{\draftcolor}{\ensuremath{n}})^{3/2}\sqrt{{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}}+\frac{90}{\sqrt{\textcolor{\draftcolor}{\kappa}\textcolor{\draftcolor}{\ensuremath{n}}^3}}\textcolor{\draftcolor}{\nu}(\log\textcolor{\draftcolor}{\ensuremath{n}})^{3/2}\sqrt{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}\biggr)\\
     &~\le~\frac{1}{2\textcolor{\draftcolor}{\ensuremath{n}}}\,.
\end{align*}
We finally absorb all the constants ($180/\sqrt{\textcolor{\draftcolor}{\kappa}}$) in $\textcolor{\draftcolor}{\nu}$   and use the definition of $\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$ to complete  the proof.
\end{proof}

\subsubsection{Proof of Lemma~\ref{derivatives}}
\begin{proof}
The proof consists of basic algebra.

\emph{Claim~1:}
We  use 1.~the definition of $\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$, 2.~the chain rule, and 3.~taking the derivatives  to obtain
\begin{align*}
  \frac{\partial}{\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}
&=\frac{\partial}{\partial\textcolor{\draftcolor}{\parameteroutE_j}}\biggl(\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\biggr)\\
&=-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\frac{\partial}{\partial\textcolor{\draftcolor}{\parameteroutE_j}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\Bigr)\\
&=-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr)\,,
\end{align*}
as desired.

\emph{Claim~2:}
We use 1.~the definition of $\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$, 2.~the chain rule, and 3.~taking the derivatives to obtain
\begin{align*}
  \frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}
&=\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\biggl(\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\biggr)\\
&=-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\Bigr)\\
&=-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\parameteroutE_j}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)\,,
\end{align*}

as desired.

\emph{Claim~3:}
We  1.~use Claim~1, and 2.~remove the term with zero derivative and use the chain rule to obtain
\begin{align*}
     \frac{\partial^2}{\partial\textcolor{\draftcolor}{\parameteroutE_{j'}}\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}&= \frac{\partial}{\partial\textcolor{\draftcolor}{\parameteroutE_{j'}}}\biggl(-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr)\biggr)\\
     &=\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j'}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\bigr)\,,
\end{align*}
as desired.

\emph{Claim~4:}
We  1.~use Claim~2, 2.~remove the term with zero derivative, and 3.~take the derivative of the bracket,
and 4.~do some rearranging to obtain
\begin{align*}
     \frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}&= \frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}}\biggl(-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\parameteroutE_j}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)\biggr)\\
     &=\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}}\biggl(\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\parameteroutE_j}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\Bigr)\biggr)\\
     &=\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\parameteroutE_j}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\parameteroutE_{j'}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k'}}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr)\\
     &=\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\parameteroutE_{j'}}\textcolor{\draftcolor}{\parameteroutE_j}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k'}}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr)\,,
\end{align*}
as desired.

\emph{Claims~5 and~6:} We only show the results for $\frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$. The results for $\frac{\partial^2}{\partial\textcolor{\draftcolor}{\parameteroutE_{j'}}\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$ can be obtained using the same arguments.

We consider two cases:

\emph{Case 1:} if $\textcolor{\draftcolor}{\ensuremath{j'}}=\textcolor{\draftcolor}{\ensuremath{j}}$, we use 1.~Claim~1, 2.~the chain rule, and 3.~taking the derivatives and simplifying
to obtain
\begin{align*}
    \frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k'}}}\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}&=\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k'}}}}\biggl(-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr)\biggr)\\
    &=-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k'}}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)+\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\theta}}_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k'}}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\biggr)\\
    &= \frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\textcolor{\draftcolor}{\parameteroutE_j}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k'}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}-\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k'}}}\Bigr)\,.
\end{align*}

\emph{Case 2:} if $\textcolor{\draftcolor}{\ensuremath{j'}}\ne \textcolor{\draftcolor}{\ensuremath{j}}$, we use 1.~Claim~1, 2.~the chain rule, and 3.~taking the derivatives an rearranging to obtain
\begin{align*}
    \frac{\partial^2}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}
    &=\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}}\biggl(-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\Bigr)\biggr)\\
    &=-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\biggl((\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)+\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{j'k'}}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\biggr)\\
    &= \frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\parameteroutE_{j'}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k'}}}(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\,,
\end{align*}

as desired.
\end{proof}
\subsubsection{Proof of Lemma~\ref{HessianPOP}}
\begin{proof}
The proof for this lemma follows the same steps as in Lemma~\ref{derivatives}, just sums are replaced by expectations and so we omit the proof.

\end{proof}

\subsubsection{Proof of Lemma~\ref{Empproc}}
\begin{proof}
We start the proof with H\"{o}lder's inequality and the definition of $\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}$, which implies $\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}~\le~\textcolor{\draftcolor}{\eta}$ for all $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}$ to obtain
\begin{align*}
     \sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}}\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-&\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}})\Bigr| \\
     &\le \sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}}\bigl( \normsupB{\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}\bigr)\\
     &\le \textcolor{\draftcolor}{\eta}\sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}} \normsupB{\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}}\,.
\end{align*}
The rest of the proof is using our  Lemma~\ref{derivatives} and  Bernstein's inequality~\citep[Corollary~2.8.3]{Vershynin2018} to find an upper bound for $\sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}} \normsup{\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}}$.
Note that for simplifying the notation, we use $\mathbb{E}[\cdot]$ as a shorthand notation of $\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_1,\textcolor{\draftcolor}{\ensuremath{y}}_1),\dots,(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_{\textcolor{\draftcolor}{\ensuremath{n}}},\textcolor{\draftcolor}{\ensuremath{y}}_{\textcolor{\draftcolor}{\ensuremath{n}}})}[\cdot]$ throughout this proof.

 We use
 1.~our result in~Lemma~\ref{derivatives} and i.i.d assumption on the data, 2.~Equation~\eqref{target} and our assumption that $\textcolor{\draftcolor}{\mathfrak{f}}[\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}]= \textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top \textcolor{\draftcolor}{\ensuremath{\Theta^*}} \textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$,  zero-mean noise, linearity of expectations, and factorizing,
 3.~the definition of sup-norm, triangle inequality,   and H\"{o}lder's inequality,
 4.~the definition of $\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}$, which implies  $\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}}\le \textcolor{\draftcolor}{\epsilon}$,
 5.~adding a zero-valued term and rewriting, and
 6.~the triangle inequality  and the definition of  $\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}$, which implies $\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}\le \normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~\textcolor{\draftcolor}{\eta}$, to obtain for each $\textcolor{\draftcolor}{\ensuremath{j}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$ and $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$ that
 \begingroup
 \allowdisplaybreaks
\begin{align*}
    &\Bigl|\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}} \Bigr|\\
    &=\biggl|-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})\textcolor{\draftcolor}{\parameteroutE_j}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}+\mathbb{E}\biggl[\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})\textcolor{\draftcolor}{\parameteroutE_j}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr]\biggr|\\
    &=2|\textcolor{\draftcolor}{\parameteroutE_j}|\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}+(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}})\bigl(\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]\bigr)\Bigr)\biggr|\\
    &\leq2\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}\biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr|+\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]-\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr)}\biggr)\\
    &\leq2\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}\biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr|+\textcolor{\draftcolor}{\epsilon}\normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]-\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\bigr)}\biggr)\\
      &= 2\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}+\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}\biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr|+\textcolor{\draftcolor}{\epsilon}\normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]\bigr)}\biggr)\\
       &\le 2(\textcolor{\draftcolor}{\eta}+\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}})\biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr|+\textcolor{\draftcolor}{\epsilon}\normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]\bigr)}\biggr)\,.
\end{align*}
\endgroup
We continue to work on the absolute value and sup-norm term in the last inequality above separately.
For each $i\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{n}}\}$ and $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$, we use our assumptions on $\textcolor{\draftcolor}{\ensuremath{\datain_i}}$ and $\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}$ to obtain that $z_i\deq\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}$ are independent and sub-exponential random variables with zero-mean~\citep[Lemma~2.7.7]{Vershynin2018} and so, we can employ Bernstein's inequality in~\citet[Corollary~2.8.3]{Vershynin2018}
to obtain for each $\textcolor{\draftcolor}{t}\in[0,\infty)$ that
\begin{align*}
   \textcolor{\draftcolor}{\mathbb{P}} \biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr|~\ge~\textcolor{\draftcolor}{t} \biggr)~\le~2e^{-\textcolor{\draftcolor}{\kappa}\min\{\textcolor{\draftcolor}{t}^2/{\textcolor{\draftcolor}{\nu}^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}}\}\textcolor{\draftcolor}{\ensuremath{n}}}
\end{align*}
with $\textcolor{\draftcolor}{\kappa}\in(0,\infty)$ an absolute constant and $\textcolor{\draftcolor}{\nu}\deq \max_{i\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{n}}\}}\norm{\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}}_{\psi_1}\in(0,\infty)$ a constant that depends on the distributions of~$\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$ and $\textcolor{\draftcolor}{u}$ (for a sub-exponential random variable $z$, we define $\norm{z}_{\psi_1}\deq \inf\{q\in(0,\infty): \mathbb{E} e^{(|z|/q)}\le 2\}$).

Now we study the behavior of the sup-norm term in the last inequality of the earlier display.
Let rewrite the sup-norm in the form of a max as
\begin{align*}
    \normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]\bigr)}=\max_{\textcolor{\draftcolor}{\ensuremath{k}}'\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]\bigr)\biggr|\,.
\end{align*}
Following the same argument as earlier and for each $i\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{n}}\}$ and  $\textcolor{\draftcolor}{\ensuremath{k}},\textcolor{\draftcolor}{\ensuremath{k}}'\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$, we use our assumption on $\textcolor{\draftcolor}{\ensuremath{\datain_i}}$  to obtain that $z'_i\deq(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]$  are independent sub-exponential random variables with zero-mean and  again we can employ Bernstein's inequality~\citep[Corollary~2.8.3]{Vershynin2018}
to obtain for each  $\textcolor{\draftcolor}{t}'\in[0,\infty)$ that
\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}\Biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]\bigr)\biggr|~\ge~\textcolor{\draftcolor}{t}'\Biggr)~\le~2e^{-\textcolor{\draftcolor}{\kappa}'\min\{\textcolor{\draftcolor}{t}'^2/{\textcolor{\draftcolor}{\nu}'^2},\textcolor{\draftcolor}{t}'/{\textcolor{\draftcolor}{\nu}'}\}\textcolor{\draftcolor}{\ensuremath{n}}}
\end{align*}
with $\textcolor{\draftcolor}{\kappa}'\in(0,\infty)$  an absolute constant and $\textcolor{\draftcolor}{\nu}'\deq~\max_{i\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{n}}\}}\norm{(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]}_{\psi_1}\in(0,\infty)$ a constant  that depends on the distribution of~$\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$.

Then, we use our result above together with the fact that if  $\textcolor{\draftcolor}{\mathbb{P}}(|b_i|~\ge~ \textcolor{\draftcolor}{t})~\le~a$ holds for all $i\in\{1,\dots p\}$, then we also have $\textcolor{\draftcolor}{\mathbb{P}}(\max_{i\in\{1,\dots p\}} |b_i|~\ge~ \textcolor{\draftcolor}{t})~\le~p a$ to obtain

\begin{equation*}
    \textcolor{\draftcolor}{\mathbb{P}}\Biggl(\max_{\textcolor{\draftcolor}{\ensuremath{k}}'\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}-\mathbb{E}[(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}]\bigr)\biggr|~\ge~\textcolor{\draftcolor}{t}'\Biggr)
    \le2\textcolor{\draftcolor}{\ensuremath{d}} e^{-\textcolor{\draftcolor}{\kappa}'\min\{\textcolor{\draftcolor}{t}'^2/{\textcolor{\draftcolor}{\nu}'^2},\textcolor{\draftcolor}{t}'/{\textcolor{\draftcolor}{\nu}'}\}\textcolor{\draftcolor}{\ensuremath{n}}}\,.
\end{equation*}
Collecting all pieces  above together with considering $\textcolor{\draftcolor}{t}=\textcolor{\draftcolor}{t}'$, we obtain for each $\textcolor{\draftcolor}{\ensuremath{j}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$ and $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$ that \begin{align*}
    \Bigl|\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\frac{\partial}{\partial\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}} \Bigr|~\le~2\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\eta}+\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}})(1+\textcolor{\draftcolor}{\epsilon})
\end{align*}
 with probability at least $1-2e^{-\textcolor{\draftcolor}{\kappa}\min\{{\textcolor{\draftcolor}{t}^2}/{\textcolor{\draftcolor}{\nu}^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}}\}\textcolor{\draftcolor}{\ensuremath{n}}}-2\textcolor{\draftcolor}{\ensuremath{d}} e^{-\textcolor{\draftcolor}{\kappa}'\min\{{\textcolor{\draftcolor}{t}^2}/{\textcolor{\draftcolor}{\nu}'^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}'}\}\textcolor{\draftcolor}{\ensuremath{n}}}$, which is obtained using the fact that
 \begin{equation*}
     P(A+bD~\le~t + bt)=1-P(A+bD~>~t + bt)~\ge~1- P(A~>~t)-P(D~>~t)
 \end{equation*}
 for any $b\in(0,\infty)$ and $t\in\textcolor{\draftcolor}{\mathbb{R}}$.

Then, we follow the same argument as earlier and
use 1.~our result in~Lemma~\ref{derivatives} and i.i.d assumption on the data,
 2.~the properties of absolute values and linearity of expectations,
 3.~some rewriting,
 4.~H\"{o}lder's inequality,
 5.~Equation~\eqref{target} and our assumptions that $\textcolor{\draftcolor}{\mathfrak{f}}[\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}]= \textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top \textcolor{\draftcolor}{\ensuremath{\Theta^*}} \textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$,  zero-mean noise, and definition of sup-norm,
 6.~triangle inequality, compatible norms (for a matrix $A\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}\times\textcolor{\draftcolor}{\ensuremath{d}}}$, we define $\normoneInfM{A}\deq \max_{\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}'=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}|A_{\textcolor{\draftcolor}{\ensuremath{k}}',\textcolor{\draftcolor}{\ensuremath{k}}}|$)), and  the definition of  $\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}$, which implies $\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}}\le \textcolor{\draftcolor}{\epsilon}$,
 7.~adding a zero-valued term,
 8.~the triangle inequality  and the definition of $\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}$, which implies   $\normone{\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\le \normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}}~\le~\textcolor{\draftcolor}{\eta}$ to obtain for each $\textcolor{\draftcolor}{\ensuremath{j}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$  that

  \begingroup
 \allowdisplaybreaks
 \begin{align*}
     &\Bigl|\frac{\partial}{\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\frac{\partial}{\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}} \Bigr|\\
     &=\biggl|-\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\bigr)+\mathbb{E}\biggl[\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\bigr)\biggr]\biggr|\\
      &=\biggl|\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}-\mathbb{E}[(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})(\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{j}}}]\bigr)\biggr|\\
      &=\biggl|\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})\textcolor{\draftcolor}{\ensuremath{\datain_i}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\ensuremath{j}},\cdot}-\mathbb{E}[(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})\textcolor{\draftcolor}{\ensuremath{\datain_i}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\ensuremath{j}},\cdot}]\bigr)\biggr|\\
      &\leq\normsupBBB{\frac{2}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl((\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})\textcolor{\draftcolor}{\ensuremath{\datain_i}}^\top-\mathbb{E}[(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}})\textcolor{\draftcolor}{\ensuremath{\datain_i}}^\top]\bigr)}\normone{\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\ensuremath{j}},\cdot}}\\
        &=2\normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta}}}\biggl(\normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}+(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}})(\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top])\bigr)}\\
        &\le2\normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta}}}\biggl(\normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}}+\textcolor{\draftcolor}{\epsilon}\normoneInfMBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top])}\biggr)\\
        &\le2\normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\ensuremath{\Theta^*}}+\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\biggl(\normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}}+\textcolor{\draftcolor}{\epsilon}\normoneInfMBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top])}\biggr)\\
        &\le2(\textcolor{\draftcolor}{\eta}+\normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}})\biggl(\normsupBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}}+\textcolor{\draftcolor}{\epsilon}\normoneInfMBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}(\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top])}\biggr)\,.
\end{align*}
\endgroup
Then, we use the same argument as earlier to treat the sup-norm terms above (we use our assumptions on $\textcolor{\draftcolor}{\ensuremath{\datain_i}}$ and $\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}$ and application of Bernstein's inequality) to obtain that
\begin{align*}
   \Bigl|\frac{\partial}{\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\frac{\partial}{\partial\textcolor{\draftcolor}{\parameteroutE_j}}\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}} \Bigr| \le2\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\eta}+\normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}})(1+\textcolor{\draftcolor}{\epsilon})
\end{align*}
 with probability at least $1-2\textcolor{\draftcolor}{\ensuremath{d}} e^{-\textcolor{\draftcolor}{\kappa}\min\{{\textcolor{\draftcolor}{t}^2}/{\textcolor{\draftcolor}{\nu}^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}}\}\textcolor{\draftcolor}{\ensuremath{n}}}-2\textcolor{\draftcolor}{\ensuremath{d}}^2 e^{-\textcolor{\draftcolor}{\kappa}'\min\{{\textcolor{\draftcolor}{t}^2}/{\textcolor{\draftcolor}{\nu}'^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}'}\}\textcolor{\draftcolor}{\ensuremath{n}}}$ ($\textcolor{\draftcolor}{\kappa}$,~$\textcolor{\draftcolor}{\nu}$,~$\textcolor{\draftcolor}{\kappa}'$,~$\textcolor{\draftcolor}{\nu}'$  are constants  depending only on the distributions of the inputs and the noises).

Collecting all the pieces above, we obtain that for  each  $i\in\{1,\dots,\textcolor{\draftcolor}{p}\}$ the corresponding gradient difference  is bounded ($|(\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}})_i|~\le~2\textcolor{\draftcolor}{t}(\textcolor{\draftcolor}{\eta}+\max\{\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}, \normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\}) (1+\textcolor{\draftcolor}{\epsilon})$) with probability at least
 $1-4\textcolor{\draftcolor}{\ensuremath{d}}^2 e^{-\textcolor{\draftcolor}{\kappa}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}\min\{{\textcolor{\draftcolor}{t}^2}/{(\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}})^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}}\}\textcolor{\draftcolor}{\ensuremath{n}}}$ with $\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}\deq\max\{\textcolor{\draftcolor}{\nu},\textcolor{\draftcolor}{\nu}'\}$ and $\textcolor{\draftcolor}{\kappa}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}\deq\min\{\textcolor{\draftcolor}{\kappa},\textcolor{\draftcolor}{\kappa}'\}$ ($\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}$ and $\textcolor{\draftcolor}{\kappa}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}$ are constants depending only on the distributions of the inputs and noises).

 Now we use 1.~the definition of sup-norm  and 2.~our results above  together with our earlier argument about implying max operator (note that the gradient vector is of dimension $\textcolor{\draftcolor}{p}$) to obtain for each $\textcolor{\draftcolor}{t}\in[0,\infty)$ that
 \begin{align*}
     \sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}} \normsupB{\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}}~&=~ \sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}} \max_{i\in\{1,\dots,\textcolor{\draftcolor}{p}\}}\bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}\bigr)_i\bigr|\\
     &\le 2\textcolor{\draftcolor}{t}\bigl(\textcolor{\draftcolor}{\eta}+\max\{\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}, \normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\}\bigr) \bigl(1+\textcolor{\draftcolor}{\epsilon})
 \end{align*}
 with probability at least $1-4\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p} e^{-\textcolor{\draftcolor}{\kappa}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}\min\{{\textcolor{\draftcolor}{t}^2}/{(\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}})^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}}\}\textcolor{\draftcolor}{\ensuremath{n}}}$.

Collecting all pieces of the proof, we obtain for each $\textcolor{\draftcolor}{t}\in[0,\infty)$ that
\begin{align*}
     \sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}}\Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}\bigr)^\top&(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}})\Bigr| \\
     &\le \textcolor{\draftcolor}{\eta}\sup_{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{C}_{\eta,\radi}}}} \normsupB{\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}-\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}}\\
     &\le 2\textcolor{\draftcolor}{t}\textcolor{\draftcolor}{\eta}\bigl(\textcolor{\draftcolor}{\eta}+\max\{\normsup{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}}, \normsupM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\}\bigr) \bigl(1+\textcolor{\draftcolor}{\epsilon})
\end{align*}
 with probability at least $1-4\textcolor{\draftcolor}{\ensuremath{d}}^2\textcolor{\draftcolor}{p} e^{-\textcolor{\draftcolor}{\kappa}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}\min\{{\textcolor{\draftcolor}{t}^2}/{(\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}})^2},\textcolor{\draftcolor}{t}/{\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}}\}\textcolor{\draftcolor}{\ensuremath{n}}}$, where for the ease of notations we replace $\textcolor{\draftcolor}{\kappa}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}$  and $\textcolor{\draftcolor}{\nu}_{\textcolor{\draftcolor}{u},\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}}$ with $\textcolor{\draftcolor}{\nu}$ and $\textcolor{\draftcolor}{\kappa}$ (constants depending only on the distributions of the inputs and noises)  in the statement of the lemma.

\end{proof}

\subsubsection{Proof of Lemma~\ref{UBoundrisk}}
\begin{proof}
Main ingredients of the proof are symmetrization of probabilities~\citep[Lemma~16.1]{van2016estimation} and Bernstein's inequality~\citep[Corollary~2.8.3]{Vershynin2018}.

We note that for simplifying the notations, we use $\mathbb{E}[\cdot]$ as a shorthand notation of $\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_1,\textcolor{\draftcolor}{\ensuremath{y}}_1),\dots,(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_{\textcolor{\draftcolor}{\ensuremath{n}}},\textcolor{\draftcolor}{\ensuremath{y}}_{\textcolor{\draftcolor}{\ensuremath{n}}})}[\cdot]$ throughout this proof.

Let start the proof and use 1.~the definition of $\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$ and $\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$,  2.~the i.i.d assumption on the data  and that $\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}=\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}+\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}$, 3.~expanding the squared-terms and rearranging, and 4.~the triangle inequality to obtain
\begingroup
\allowdisplaybreaks
\begin{align*}
     \sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\bigl|\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}&-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\bigr|\\
     &=\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\Bigr)-\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)^2\Bigr]\biggr|\\
     &=\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}+\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2-\mathbb{E}\Bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}+\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\Bigr]\Bigr)\biggr|\\
     &=\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2-\mathbb{E}\Bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\Bigr]\Bigr)\\
     &~~~~~~~~~~~~~~~+2\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}-\mathbb{E}\bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\bigr]\Bigr)
     +\bigl(\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}^2-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}^2]\bigr)\biggr|\\
      &~\le~\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2-\mathbb{E}\Bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\Bigr]\Bigr)\biggr|\\
     &~~~~~~~~~~~~~~~
     +2\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}-\mathbb{E}\bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\bigr]\Bigr)\biggr|\\
     &~~~~~~~~~~~~~~~+\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}^2-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}^2]\bigr)\biggr|\,.
\end{align*}
\endgroup
Now, we continue to work on each term in the last inequality above  separately in steps:

\emph{Step~1:}
Using~\citet[Corollary~2.8.3]{Vershynin2018} together with our assumption on noise, which implies the squared of Gaussian noise is sub-exponential, we obtain for each $\Bar{\textcolor{\draftcolor}{t}}\in[0,\infty)$ that
\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}\biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}(\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}^2-\mathbb{E}[\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}^2])\biggr|~\ge~\Bar{\textcolor{\draftcolor}{t}}\biggr)~\le~2\littlespace e^{-\textcolor{\draftcolor}{\kappa}\min\{\Bar{\textcolor{\draftcolor}{t}}^2/\textcolor{\draftcolor}{\nu}^2,\Bar{\textcolor{\draftcolor}{t}}/\textcolor{\draftcolor}{\nu}\}\textcolor{\draftcolor}{\ensuremath{n}}}\,,
\end{align*}
where $\textcolor{\draftcolor}{\kappa},\textcolor{\draftcolor}{\nu}\in(0,\infty)$ are constants depending only on the distribution of the noise (let note that our constants $\textcolor{\draftcolor}{\kappa}$ and $\textcolor{\draftcolor}{\nu}$  may change from line to line in this proof, but they constantly depend just on the distribution of the inputs or noise or both).

\emph{Step~2:}
We now prepare the application of~\citet[Lemma~16.1]{van2016estimation}.  Let 1.~define $\textcolor{\draftcolor}{\mathcal{R}}^2$ and 2.~use H\"{o}lder's inequality and factorizing to obtain
\begin{align*}
    \textcolor{\draftcolor}{\mathcal{R}}^2~&\deq~\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\mathbb{E}\Bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^4\Bigr]\\
    &\le \sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\normoneB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}}^4\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\mathbb{E}\bigl[\normsup{\textcolor{\draftcolor}{\ensuremath{\datain_i}}}^4\bigr]\,.
\end{align*}
We also employ some linear algebra together with compatible norms (for a matrix $A\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}\times\textcolor{\draftcolor}{\ensuremath{d}}}$, we define $\normoneInfM{A}\deq \max_{\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}'=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}|A_{\textcolor{\draftcolor}{\ensuremath{k}}',\textcolor{\draftcolor}{\ensuremath{k}}}|$)) to obtain
\begin{align*}
    \biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\biggr|&= \biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\zeta}_i\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^\top\biggr|\\
    &=\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\bigr)\textcolor{\draftcolor}{\ensuremath{\datain_i}}\textcolor{\draftcolor}{\zeta}_i\textcolor{\draftcolor}{\ensuremath{\datain_i}}^\top\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\bigr)^\top\biggr|\\
     &\le \normsupB{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}})^2}\normoneInfMBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top}\,.
\end{align*}
Then, we use 1.~symmetrization of probabilities~\citep[Lemma~16.1]{van2016estimation} with $\textcolor{\draftcolor}{\mathcal{R}}$ as defined earlier, 2.~the display above,
3.~our assumption that $\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\normsup{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}})^2}~\le~\textcolor{\draftcolor}{\epsilon}'$ and some rearranging, 4.~the definition of $\ell_{\infty,1}$-norm for a matrix above, 5.~the fact that if $\textcolor{\draftcolor}{\mathbb{P}}(|b_i|~\ge~ \textcolor{\draftcolor}{t})~\le~a$ holds for all $i\in\{1,\dots
\textcolor{\draftcolor}{\ensuremath{d}}\}$, then we also have $\textcolor{\draftcolor}{\mathbb{P}}(\max_{i\in\{1,\dots \textcolor{\draftcolor}{\ensuremath{d}}\}} |b_i|~\ge~ \textcolor{\draftcolor}{t})~\le~\textcolor{\draftcolor}{\ensuremath{d}} a$ (for  $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$), 6.~the fact that for a vector $\boldsymbol{a}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}},~ \textcolor{\draftcolor}{\mathbb{P}}(\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}|{\boldsymbol{a}_i}|~\ge~\textcolor{\draftcolor}{t})~\le~\textcolor{\draftcolor}{\ensuremath{d}}\max_{\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}}\allowbreak\textcolor{\draftcolor}{\mathbb{P}}(|\boldsymbol{a}_{\textcolor{\draftcolor}{\ensuremath{k}}}|~\ge~\textcolor{\draftcolor}{t})$, and 7.~our assumption on $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$ (to get rid of max term)
together with \citet[Corollary~2.8.3]{Vershynin2018} to obtain for each $\textcolor{\draftcolor}{t}\in[0,\infty)$ that
\begingroup
\allowdisplaybreaks
\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}\Biggl(\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}&\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2-\mathbb{E}\Bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\Bigr]\Bigr)\biggr|~\ge~4\textcolor{\draftcolor}{\mathcal{R}}\sqrt{\frac{2\textcolor{\draftcolor}{t}}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
    &\le~4\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\biggr|~\ge~\textcolor{\draftcolor}{\mathcal{R}}\sqrt{\frac{2\textcolor{\draftcolor}{t}}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
    &\le~4\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\normsupB{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}})^2}\normoneInfMBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top}~\ge~\textcolor{\draftcolor}{\mathcal{R}}\sqrt{\frac{2\textcolor{\draftcolor}{t}}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
     &\le~4\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\normoneInfMBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i\textcolor{\draftcolor}{\ensuremath{\datain_i}}\dataini^\top}~\ge~\frac{\textcolor{\draftcolor}{\mathcal{R}}}{\textcolor{\draftcolor}{\epsilon}'}\sqrt{\frac{2\textcolor{\draftcolor}{t}}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
     &\le~4\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\max_{\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}'=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr|~\ge~\frac{\textcolor{\draftcolor}{\mathcal{R}}}{\textcolor{\draftcolor}{\epsilon}'}\sqrt{\frac{2\textcolor{\draftcolor}{t}}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
     &\le~4\textcolor{\draftcolor}{\ensuremath{d}}\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\sum_{\textcolor{\draftcolor}{\ensuremath{k}}'=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr|~\ge~\frac{\textcolor{\draftcolor}{\mathcal{R}}}{\textcolor{\draftcolor}{\epsilon}'}\sqrt{\frac{2\textcolor{\draftcolor}{t}}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
      &\le~4\textcolor{\draftcolor}{\ensuremath{d}}^2\max_{\textcolor{\draftcolor}{\ensuremath{k}}'\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}}\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}'}(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\biggr|~\ge~\frac{\textcolor{\draftcolor}{\mathcal{R}}}{\textcolor{\draftcolor}{\epsilon}'}\sqrt{\frac{2\textcolor{\draftcolor}{t}}{\textcolor{\draftcolor}{\ensuremath{n}}}}\eqd\textcolor{\draftcolor}{t}''\Biggr)\\
      &\le~8\textcolor{\draftcolor}{\ensuremath{d}}^2\littlespace e^{-\textcolor{\draftcolor}{\kappa}\min\{\textcolor{\draftcolor}{t}''^2/\textcolor{\draftcolor}{\nu}^2,\textcolor{\draftcolor}{t}''/\textcolor{\draftcolor}{\nu}\}\textcolor{\draftcolor}{\ensuremath{n}}}\,,
\end{align*}
\endgroup
where $\textcolor{\draftcolor}{\kappa},\textcolor{\draftcolor}{\nu}\in(0,\infty)$ are constants depending only on the distribution of the inputs.

Collecting results above, we obtain  for each $\textcolor{\draftcolor}{t}''\in[0,\infty)$ that
\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}\Biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2&-\mathbb{E}\Bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2\Bigr]\Bigr)\biggr|~\ge~4\textcolor{\draftcolor}{\epsilon}' \textcolor{\draftcolor}{t}''\Biggr)\\
    &~\le~8\textcolor{\draftcolor}{\ensuremath{d}}^2\littlespace e^{-\textcolor{\draftcolor}{\kappa}\min\{\textcolor{\draftcolor}{t}''^2/\textcolor{\draftcolor}{\nu}^2,\textcolor{\draftcolor}{t}''/\textcolor{\draftcolor}{\nu}\}\textcolor{\draftcolor}{\ensuremath{n}}}\,.
\end{align*}

\emph{Step~3:}
Let define $(\textcolor{\draftcolor}{\mathcal{R}}')^2$ and use H\"{o}lder's inequality to obtain
\begin{align*}
    (\textcolor{\draftcolor}{\mathcal{R}}')^2~&\deq~\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\mathbb{E}\Bigl[\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\Bigr)^2\Bigr]\\
    &\le \sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\normoneB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}}^2\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\mathbb{E}\bigl[\normsup{\textcolor{\draftcolor}{\ensuremath{\datain_i}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}}^2\bigr]\,.
\end{align*}
Then, we use 1.~symmetrization of probabilities~\citep[Lemma~16.1]{van2016estimation} with $\textcolor{\draftcolor}{\mathcal{R}}'$ defined as above, 2.~H\"{o}lder's inequality, 3.~our assumption that $\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\normsup{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}})^2}~\le~\textcolor{\draftcolor}{\epsilon}'$,  the fact that for a vector $\boldsymbol{a}\in\textcolor{\draftcolor}{\mathbb{R}}^d$, $\textcolor{\draftcolor}{\mathbb{P}}(\normone{\boldsymbol{a}}~\ge~t)~\le~d\max_{i\in\{1,\dots,d\}}\textcolor{\draftcolor}{\mathbb{P}}(|{\boldsymbol{a}_i}|~\ge~t)~\le~d^2\textcolor{\draftcolor}{\mathbb{P}}(|{\boldsymbol{a}_i}|~\ge~t)$,  and the assumption on inputs (for $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$), and 4.~\citet[Corollary~2.8.3]{Vershynin2018} together with our assumptions on the input and noise to obtain for each $\textcolor{\draftcolor}{t}'\in[0,\infty)$ that
\begingroup
\allowdisplaybreaks
\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}\Biggl(\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}&\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}-\mathbb{E}\Bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\Bigr]\Bigr)\biggr|~\ge~4\textcolor{\draftcolor}{\mathcal{R}}'\sqrt{\frac{2\textcolor{\draftcolor}{t}'}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
    &\le~4\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\biggr|~\ge~\textcolor{\draftcolor}{\mathcal{R}}'\sqrt{\frac{2\textcolor{\draftcolor}{t}'}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
    &\le~4\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\normsupB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}}\normoneBBB{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i\textcolor{\draftcolor}{\ensuremath{\datain_i}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}}~\ge~\textcolor{\draftcolor}{\mathcal{R}}'\sqrt{\frac{2\textcolor{\draftcolor}{t}'}{\textcolor{\draftcolor}{\ensuremath{n}}}}\Biggr)\\
     &\le~4\textcolor{\draftcolor}{\ensuremath{d}}^2~\textcolor{\draftcolor}{\mathbb{P}}\Biggl(\biggl|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\textcolor{\draftcolor}{\zeta}_i(\textcolor{\draftcolor}{\ensuremath{\datain_i}})_{\textcolor{\draftcolor}{\ensuremath{k}}}\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\biggr|~\ge~\textcolor{\draftcolor}{\mathcal{R}}'\sqrt{\frac{2\textcolor{\draftcolor}{t}'}{\textcolor{\draftcolor}{\epsilon}'\textcolor{\draftcolor}{\ensuremath{n}}}}\eqd \textcolor{\draftcolor}{t}'''\Biggr)\\
     &\le~8\textcolor{\draftcolor}{\ensuremath{d}}^2\littlespace e^{-\textcolor{\draftcolor}{\kappa}\min\{\textcolor{\draftcolor}{t}'''^2/\textcolor{\draftcolor}{\nu}^2,\textcolor{\draftcolor}{t}'''/\textcolor{\draftcolor}{\nu}\}\textcolor{\draftcolor}{\ensuremath{n}}}\,,
\end{align*}
\endgroup
where $\textcolor{\draftcolor}{\kappa},\textcolor{\draftcolor}{\nu}\in(0,\infty)$ are constants depending only on the distributions  of the inputs and noise.

Collecting results above we obtain that
\begin{align*}
    \textcolor{\draftcolor}{\mathbb{P}}\Biggl(\biggr|\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\Bigl(\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}-&\mathbb{E}\bigl[\bigl(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}\bigr]\Bigr)\biggr|~\ge~4 \sqrt{\textcolor{\draftcolor}{\epsilon}'}\textcolor{\draftcolor}{t}'''\Biggr)\\
    &~\le~8\textcolor{\draftcolor}{\ensuremath{d}}^2\littlespace e^{-\textcolor{\draftcolor}{\kappa}\min\{\textcolor{\draftcolor}{t}'''^2/\textcolor{\draftcolor}{\nu}^2,\textcolor{\draftcolor}{t}'''/\textcolor{\draftcolor}{\nu}\}\textcolor{\draftcolor}{\ensuremath{n}}}\,,
\end{align*}
where $\textcolor{\draftcolor}{\kappa},\textcolor{\draftcolor}{\nu}\in(0,\infty)$ are constants depending only on the distributions of the inputs and noise.

Collecting all the pieces of the proof  in steps~1:3,  we obtain for each $\textcolor{\draftcolor}{t}\in[0,\infty)$ that
\begin{align*}
     \sup_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\bigl|\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}-\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\bigr|~&\le~\textcolor{\draftcolor}{t}\bigl(1+4\textcolor{\draftcolor}{\epsilon}'+4\sqrt{\textcolor{\draftcolor}{\epsilon}'}\bigr)
\end{align*}
with probability at least $1-(2+8\textcolor{\draftcolor}{\ensuremath{d}}^2+8\textcolor{\draftcolor}{\ensuremath{d}}^2)\littlespace e^{-\textcolor{\draftcolor}{\kappa}\min\{\textcolor{\draftcolor}{t}^2/\textcolor{\draftcolor}{\nu}^2,\textcolor{\draftcolor}{t}/\textcolor{\draftcolor}{\nu}\}\textcolor{\draftcolor}{\ensuremath{n}}}$ or $1-18\textcolor{\draftcolor}{\ensuremath{d}}^2\littlespace e^{-\textcolor{\draftcolor}{\kappa}\min\{\textcolor{\draftcolor}{t}^2/\textcolor{\draftcolor}{\nu}^2,\textcolor{\draftcolor}{t}/\textcolor{\draftcolor}{\nu}\}\textcolor{\draftcolor}{\ensuremath{n}}}$ (using the assumption that $\textcolor{\draftcolor}{\ensuremath{d}}~\ge~1$), where we consider $\textcolor{\draftcolor}{t}=\Bar{\textcolor{\draftcolor}{t}}=\textcolor{\draftcolor}{t}''=\textcolor{\draftcolor}{t}'''$ and $\textcolor{\draftcolor}{\kappa},\textcolor{\draftcolor}{\nu}\in(0,\infty)$ are constants depending only on the distributions of the inputs and noise.
\end{proof}
\subsubsection{Proof of Lemma~\ref{Inv_M}}
\begin{proof}
The proof follows just basic linear algebra.

Since $H(t)$ is invertible exactly when $(A+tC)^\top$  have full (column) rank, we are left to study the rank of $(A+tC)^\top=A^\top+tC^\top$.
To do so, we employ the Singular Value Decomposition (SVD) of $A^T\in \textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}'\times\textcolor{\draftcolor}{\ensuremath{w}}'}$,
that is, $A^\top=UDV^\top$ with $U\in \textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}'\times\textcolor{\draftcolor}{\ensuremath{w}}'}$, $V\in \textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}'\times \textcolor{\draftcolor}{\ensuremath{w}}'}$, and $D\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}'\times\textcolor{\draftcolor}{\ensuremath{w}}'}$ that $U,V$ are semi-orthogonal matrices and $D$ has the same  rank as $A$, in this case full rank.
Now, we are motivated to make a squared matrix as
\begin{equation*}
    U^\top(A^\top+tC^\top)V=U^\top(UDV^\top+tC^\top)V=D+tU^\top C^\top V=tD(t^{-1}I_{\textcolor{\draftcolor}{\ensuremath{w}}'}+D^{-1}U^\top C^\top V)\,,
\end{equation*}
where we used the SVD form of matrix $A$, orthogonal property of $U,V$, and some rewriting.
Since matrices $U$ and $V$ have rank $\textcolor{\draftcolor}{\ensuremath{w}}$, for studying the rank of $A^\top+tC^\top$ it is enough to study determinant  of $U^\top(A^\top+tC^\top)V$.
We then use our display above, properties of determinants for squared matrices, and characteristic polynomial  to obtain
\begin{align*}
  \det\bigl(U^\top(A^\top+tC^\top\bigr)V)&=\det\bigl(tD(t^{-1}I_{\textcolor{\draftcolor}{\ensuremath{w}}'}+D^{-1}U^\top C^\top V)\bigr)\\
  &=\det(tD)\det\bigl(t^{-1}I_{\textcolor{\draftcolor}{\ensuremath{w}}'}+D^{-1}U^\top C^\top V\bigr)\\
  &=t^{\textcolor{\draftcolor}{\ensuremath{w}}'}\det(D)p_{Z \deq D^{-1}U^\top C^\top V}(-t^{-1})\,.
\end{align*}
Since, $\det(D)\ne 0$ and $t\ne 0$, then the $t$ which $H(t)$ is singular are the roots of $p_{Z}(-t^{-1})$, where $Z=D^{-1}U^\top C^\top V$.
Since the roots of $p_{Z}$ are the eigenvalues of $Z$, we have found that the only $t$ for which $H(t)$ fails to be invertible are the negative reciprocals of the (nonzero) eigenvalues of $Z$.
Since, any $\textcolor{\draftcolor}{\ensuremath{w}}'\times\textcolor{\draftcolor}{\ensuremath{w}}'$ matrix has at most $\textcolor{\draftcolor}{\ensuremath{w}}'$ distinct eigenvalues, there are just finitely many $t$ such that $H(t)$ is not invertible, as desired.
\end{proof}

 \begin{figure}
 	\centering
 	\begin{subfigure}[t]{2.80in}
 		\centering
 		\includegraphics[width=7cm, height=5cm]{Figures/Optim_Linear.png}

 	\end{subfigure}
 	\quad
 	\begin{subfigure}[t]{2.80in}
 		\centering
 		\includegraphics[width=7cm, height=5cm]{Figures/Optim_Relu.png}

 	\end{subfigure}
 	\caption{Training error   for trained  neural networks (with $\textcolor{\draftcolor}{\ensuremath{d}}=\textcolor{\draftcolor}{\ensuremath{w}}=10$)  with   linear (left panel) and ReLU (right panel) activations in $5$ different runs (allocated with different colors).   Due to the non-convexity of neural networks,  optimization algorithm may end up in different approximate stationary points.}
	\label{fig:Optim_Linear&Relu}
 \end{figure}

\section{Introduction}
Statistical theories for deep learning usually apply to exact, global optima of certain objective functions \citep{Bartlett1998,Bartlett2002,bauer2019deep,kohler2021rate,Lederer2020,taheri2021,schmidt2020}.
But those objective functions cannot be solved explicitly and are highly non-convex,
so that in practice, exact, global optimization is---at least to date---out of reach,
and we can expect only approximate stationary points from current algorithms (see Figure~\ref{fig:minima}).
In other words,
it is unclear whether the known theories have any meaning for the outputs of actual deep-learning pipelines.
\begin{figure

Also other parts of machine learning face optimization problems that are challenging to optimize globally and to full precision.
Accordingly, some statistical insights have already been established.
For example,
\citet{bien2018non,bien2019prediction} solve a non-convex problem in linear regression in a ``convex'' way and develop statistical theories for their solution.
\citet{loh2015,loh2017statistical} develop statistical theory for stationary points in another regression setup under curvature assumptions.
\citet{elsener2018sharp} establish more general theories for stationary points also under curvature assumptions.
\citet{Taheri2019} propose an algorithm and statistical theory for approximate solutions in a convex setting.
But it is currently unclear how to extend these insights to deep learning---if at all possible.

In this paper,
we develop statistical guarantees for the stationary points of simple neural networks (Theorem~\ref{OracleInSta}) and for the points in the vicinity of a stationary point (Theorem~\ref{StaAppr}).
Strikingly, the statistical rates match the ``theoretical'' rates in~\cite{taheri2021} up to log-terms.
Hence, although our theories here are limited to simple networks, our paper makes a significant step toward the mathematical understanding of practical neural-network pipelines.

\paragraph{Motivation}
Let us briefly motivate our study in a very simple example.
Consider regression data consisting of the two input-output pairs $(\textcolor{\draftcolor}{\ensuremath{x}}_1,\textcolor{\draftcolor}{\ensuremath{y}}_1)=(-1,0)$ and  $(\textcolor{\draftcolor}{\ensuremath{x}}_2,\textcolor{\draftcolor}{\ensuremath{y}}_2)=(1,1)$.
On the one hand, we can relate the inputs and outputs through a simple linear-regression model
$\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}= \omega^*\textcolor{\draftcolor}{\ensuremath{x}}_i+\textcolor{\draftcolor}{\ensuremath{u}}_i$ parameterized by $\omega^*\in\textcolor{\draftcolor}{\mathbb{R}}$.
The least-squares estimate for~$\omega^*$ minimizes the objective function~$\mathfrak{f}_1\,:\,\omega\mapsto \omega^2+(1-\omega)^2$.
The function $\mathfrak{f}_1$ is benign;
in particular, it is smooth and convex (see the left panel of Figure~\ref{fig:ConVsNonCon}).

On the other hand,
we can also relate the inputs and outputs through a very simple linear neural-network model $\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}=\gamma^*\theta^*\textcolor{\draftcolor}{\ensuremath{x}}_i+\textcolor{\draftcolor}{\ensuremath{u}}_i$ parametrized by $\gamma^*,\theta^*\in\textcolor{\draftcolor}{\mathbb{R}}$.
The least-squares estimate for $\gamma^*,\theta^*$ minimizes the objective function~$\mathfrak{f}_2\,:\,(\gamma,\theta)\mapsto(\gamma\theta)^2+(1-\gamma\theta)^2$.
The earlier estimate and the one here are equivalent from a statistical perspective:
we can simply identify~$\omega$ with the product~$\gamma\theta$.
But the reparametrization changes the computational properties:
the function $\mathfrak{f}_2$ is not benign any more; in particular, it is not convex (see the right panel of Figure~\ref{fig:ConVsNonCon}).

This simple example illustrates that going from linear regression and other standard models to neural networks naturally leads to computational challenges;
in particular,
the corresponding objective functions are usually non-convex,
which means that
optimization algorithms may easily get stuck in non-optimal stationary points.
Thus,
we argue that statistical guarantees in deep learning should concern stationary points rather than just global optima.

\begin{figure}[t]
    \centering
    \scalebox{1}{
    \includegraphics[width=7cm, height=5cm]{Figures/convex.png}}
    \scalebox{1}{
    \includegraphics[width=7cm, height=5cm]{Figures/Nonconvex.png}}
    \caption{Objective function of least-squares in linear regression (left panel) and in a neural-network model (right panel) for data  $(\textcolor{\draftcolor}{\ensuremath{x}}_1,\textcolor{\draftcolor}{\ensuremath{y}}_1)=(-1,0)$ and  $(\textcolor{\draftcolor}{\ensuremath{x}}_2,\textcolor{\draftcolor}{\ensuremath{y}}_2)=(1,1)$.
    The linear-regression function is convex while the neural-network function is highly non-convex.}
    \label{fig:ConVsNonCon}
\end{figure}

\paragraph{Paper outline}
In Section~\ref{Mainresult}, we  state the statistical guarantees for the stationary points of the neural-network objectives (Theorem~\ref{OracleInSta}) and the statistical guarantees for approximations of a stationary point (Theorem~\ref{StaAppr}).
We provide some  numerical observations in Section~\ref{NumSec}.
In Section~\ref{TechResult}, we study the behavior of the Hessian matrix of the population risk (Proposition~\ref{PSDHessian}) and then employ tools from empirical-process theory to bound the absolute  difference between the gradient vector of the prediction and population risk (Lemma~\ref{EmpProcctwo});
these are the main ingredients for proving our Theorem~\ref{OracleInSta}.
In Section~\ref{RelatedWork}, we provide an overview of the related works.
More technical results and detailed proofs are  deferred to the Section~\ref{MoreResult}.
We conclude our paper in Section~\ref{discussion}.

\paragraph{Notations}
We use $\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})$ to generate a vector of length $\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}+\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}}}$ from a vector $\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$ and a matrix $\textcolor{\draftcolor}{\ensuremath{\Theta}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times\textcolor{\draftcolor}{\ensuremath{d}}}$ (for generating the vector, we first push the elements of $\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}$ and then elements of $\textcolor{\draftcolor}{\ensuremath{\Theta}}$ row by row).
We collect first order partial derivatives of prediction risk $\objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$ and population risk  $\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$ with respect to the $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}\deq\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})$ in the gradient vectors $\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveF{\parameterout,\parameterin}}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}+\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}}}$ and $\textcolor{\draftcolor}{\ensuremath{\nabla\objectiveFP{\parameterout,\parameterin}}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}+\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}}}$, respectively.
We use the notation $\norm{\cdot}$ for a general vector norm and $\normM{\cdot}$ for a  general matrix norm.
We also define $\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}\deq\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}|\textcolor{\draftcolor}{\gamma}_{\textcolor{\draftcolor}{\ensuremath{j}}}|$ and $\normoneM{\textcolor{\draftcolor}{\ensuremath{\Theta}}}\deq\sum_{\textcolor{\draftcolor}{\ensuremath{j}}=1}^{\textcolor{\draftcolor}{\ensuremath{w}}}\sum_{\textcolor{\draftcolor}{\ensuremath{k}}=1}^{\textcolor{\draftcolor}{\ensuremath{d}}}|\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}|$.

\section{Main results}\label{Mainresult}
Consider inputs $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_{\textcolor{\draftcolor}{\ensuremath{n}}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}}$ and corresponding outputs $\textcolor{\draftcolor}{\ensuremath{y}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{y}}_{\textcolor{\draftcolor}{\ensuremath{n}}}\in\textcolor{\draftcolor}{\mathbb{R}}$ that are connected via
\begin{equation}\label{target}
    \textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}~=~\textcolor{\draftcolor}{\TrueFun[\dataini]}+\textcolor{\draftcolor}{\ensuremath{\noiseE_i}}
\end{equation}
for an unknown target function $\textcolor{\draftcolor}{\mathfrak{f}}\,:\,\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}}\to\textcolor{\draftcolor}{\mathbb{R}}$ and unknown stochastic noise $\textcolor{\draftcolor}{u}_1,\dots,\textcolor{\draftcolor}{u}_{\textcolor{\draftcolor}{\ensuremath{n}}}\in\textcolor{\draftcolor}{\mathbb{R}}$.
Deep learning is about using the available data to approximate the unknown target function~$\textcolor{\draftcolor}{\mathfrak{f}}$ by a neural network.
In this paper, we consider shallow and linear neural networks of the form
\begin{equation*}
    \textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}~\mapsto~\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\,,
\end{equation*}
where
\begin{equation*}
    (\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})~\in~\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}\,\deq\,\bigl\{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})~\in~\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}\times\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\times\textcolor{\draftcolor}{\ensuremath{d}}}\bigr\}\,.
\end{equation*}
We assume that the target function can be approximated by such a neural network in the first place:
there is a pair $ (\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}$
such that $\textcolor{\draftcolor}{\mathfrak{f}}[\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}]~\approx~\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top \textcolor{\draftcolor}{\ensuremath{\Theta^*}} \textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$.
In fact, to have a sharp focus on  statistical guarantees rather than the approximation properties of neural networks, we simply assume that  $\normone{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}},\normoneM{\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}\leq \sqrt{\log{\textcolor{\draftcolor}{\ensuremath{n}}}}$
and $\textcolor{\draftcolor}{\mathfrak{f}}[\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}]=\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top \textcolor{\draftcolor}{\ensuremath{\Theta^*}} \textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}$ for all $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{d}}}$,
that is,
the target function is itself a neural network with reasonably small parameters.\footnote{The bound $\sqrt{\log{\textcolor{\draftcolor}{\ensuremath{n}}}}$ here and on the next page is merely for convenience:
it can be replaced by any fixed constant or another function that is increasing mildly in the sample size~$\textcolor{\draftcolor}{\ensuremath{n}}$.}
Note that parametrization  of neural networks is ambiguous:
 there are typically infinitely many pairs $ (\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}$ that satisfy those conditions---compare to~\citet[Proposition~1]{taheri2021};
for further reference,
we define $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}\deq\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}})$ for a fixed but arbitrary such pair of parameters.
Note that the concentration of this study is on linear neural networks. Although linear neural networks are not used in practice,   theoretical understanding of them is important because, they may  reveal some properties of non-linear models (see our Section~\ref{NumSec}), while theoretical study of them is more simple.

The network parameters in deep learning are usually estimated via standard statistical techniques, such as least-squares or logistic loss, complemented by some type of regularization, such as dropout \citep{Hinton2012, Molchanov2017,Neklyudov2017,Salehinejad2019},
batch normalization~\citep{ioffe2015batch},
explicit regularization  \citep{Alvarez16,Feng17,Liu15,Scardapane17},
or low-rank approximation \citep{Denil2013,Denton2014,Jaderberg2014}.
To fix ideas,
we focus on least-squares complemented by (elementwise) $\ell_1$-regularization:
\begin{equation}\label{ls}
(\textcolor{\draftcolor}{\widehat\parameterout},\textcolor{\draftcolor}{\widehat\parameterin})~\in~\ensuremath{\arg\min}_{(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}}\biggl\{\frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})}\biggr\}\,,
\end{equation}
where $\textcolor{\draftcolor}{\ensuremath{r}}\in[0,\infty)$ is a tuning parameter to be calibrated (see~\cite{sardy2020needles} for some theory insights).
Such estimators
are standard in machine learning and statistics~\citep{Lederer2021HD,eldar2012compressed}.
As usual, we measure the  (in-sample-)prediction risk by
\begin{equation*}
    \objectiveF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}~\deq~ \frac{1}{\textcolor{\draftcolor}{\ensuremath{n}}}\sum_{i=1}^{\textcolor{\draftcolor}{\ensuremath{n}}}\bigl(\textcolor{\draftcolor}{\ensuremath{\dataoutE_i}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\datain_i}}\bigr)^2
\end{equation*}
with $\textcolor{\draftcolor}{\ensuremath{X}}\deq(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_{\textcolor{\draftcolor}{\ensuremath{n}}})^\top\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{n}}\times\textcolor{\draftcolor}{\ensuremath{d}}}$ and the generalization risk by
\begin{equation*}
    \ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}]~\deq~\mathbb{E}_{(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})}\Bigl[\bigl(\textcolor{\draftcolor}{\ensuremath{y}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}\bigr)^2\Bigr]
\end{equation*}
with the expectation over a new sample $(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})$ (that has the same distribution as $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_{\textcolor{\draftcolor}{\ensuremath{n}}}$ and $\textcolor{\draftcolor}{\ensuremath{y}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{y}}_{\textcolor{\draftcolor}{\ensuremath{n}}}$).
We call $\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}\deq\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin})$  a \emph{stationary point} of the objective in~\eqref{ls} if it satisfies~\citep[Equation~6]{elsener2018sharp};\citep[Equation~5]{loh2015}
\begin{equation}\label{StaP}
   \bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})+\textcolor{\draftcolor}{\ensuremath{r}}\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})~\ge~ 0~~~~~\forall~ \textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})~~ \text{with}~~(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in~\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}
\end{equation}
for appropriate $\textcolor{\draftcolor}{\tilde{\boldsymbol{z}}}\in\boldsymbol{\partial}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}$ (where $\boldsymbol{\partial}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}$ is the subdifferential of the regularizer at $\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}$).
We call a stationary point $
\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}$ \emph{reasonable} once $\normone{\textcolor{\draftcolor}{\widetilde\parameterout}},\normoneM{\textcolor{\draftcolor}{\widetilde\parameterin}}\leq \sqrt{\log{\textcolor{\draftcolor}{\ensuremath{n}}}}$---again to avoid unnecessary complication.
Due to the ambiguity of neural networks, there are  infinitely many equivalent stationary and reasonable stationary points;
importantly, our guarantees hold for every (reasonable) stationary point and target~$\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}$.

We say that a network indexed by $(\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin})$ generalizes well if
\begin{equation*}
   \ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}]~\approx\,\operatorname{risk}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]\,,
\end{equation*}
that is, the network generalizes essentially as well as the best network.
In the following, we show that not only the ``statistical'' network indexed by $(\textcolor{\draftcolor}{\widehat\parameterout},\textcolor{\draftcolor}{\widehat\parameterin})$ but also every ``practical'' network indexed by a reasonable stationary point $(\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin})$ of the objective function in~\eqref{ls} generalizes well.
For simplicity, we assume that $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_{\textcolor{\draftcolor}{\ensuremath{n}}}\in\textcolor{\draftcolor}{\mathcal{N}(\boldsymbol{0},\ID)}$ are i.i.d.~standard Gaussian random vectors
and that $\textcolor{\draftcolor}{\ensuremath{u}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{u}}_{\textcolor{\draftcolor}{\ensuremath{n}}}\in\textcolor{\draftcolor}{\mathcal{N}(0,\StandDeviEr^2)}$ are centered and Gaussian random variables and
independent of $\textcolor{\draftcolor}{\ensuremath{\datain_i}}$.
Moreover,
we call the total number of parameters in the network $\textcolor{\draftcolor}{p}\deq\textcolor{\draftcolor}{\ensuremath{w}}+\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}}$ the problem's effective dimension and
\begin{equation}
    \textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}~\deq~\textcolor{\draftcolor}{\nu}(\log{\textcolor{\draftcolor}{\ensuremath{n}}})^{3/2}\sqrt{\frac{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}{\textcolor{\draftcolor}{\ensuremath{n}}}}
    \end{equation}
the oracle tuning parameter, where $\textcolor{\draftcolor}{\nu}\in(0,\infty)$ is a constant that depends only on
the distributions of the inputs and noises.
It has been shown that $\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$ is indeed an optimal tuning parameter of~\eqref{ls} in some sense~\citep{taheri2021}.

We then get the following result for a new sample pair $(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}},\textcolor{\draftcolor}{\ensuremath{y}})$ with the same distribution as $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\datainE}}}_{\textcolor{\draftcolor}{\ensuremath{n}}}$ and $\textcolor{\draftcolor}{\ensuremath{y}}_1,\dots,\textcolor{\draftcolor}{\ensuremath{y}}_{\textcolor{\draftcolor}{\ensuremath{n}}}$.

\begin{theorem}[Statistical Guarantees for All Reasonable Stationary Points]\label{OracleInSta}
Any  reasonable stationary point $(\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin})$
of the objective function in~\eqref{ls} with $\textcolor{\draftcolor}{\ensuremath{r}}\ge \textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$ satisfies the risk bound
\begin{equation}\label{main_ineq}
 \ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}]~\leq~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+5\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}
 \end{equation}
with probability at least
$1-1/2\textcolor{\draftcolor}{\ensuremath{n}}$.
If $\textcolor{\draftcolor}{\ensuremath{r}}=\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$,
the bound becomes
\begin{equation}
 \ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}]~\leq~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+\textcolor{\draftcolor}{\nu}(\log{\textcolor{\draftcolor}{\ensuremath{n}}})^{2}\sqrt{\frac{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}{\textcolor{\draftcolor}{\ensuremath{n}}}}\,.
\end{equation}
\end{theorem}
The proof of Theorem~\ref{OracleInSta} is given in Section~\ref{proofs}.

Theorem~\ref{OracleInSta} proves the fact that for properly chosen tuning parameter~$\textcolor{\draftcolor}{\ensuremath{r}}$ and large enough sample sizes,
any reasonable stationary point of~\eqref{ls} generalizes essentially as well as $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}$.
Our results essentially have the same rates as the ones in the literature~\citep[Theorem~3]{taheri2021} but, in stark contrast to those results, apply to all reasonable stationary points (including saddle points) rather than to the global optimum of the objective function only.

The assumption of Gaussian noise can be relaxed readily by replacing the Gaussian-type results in the empirical-process derivations by more general results~\citep{adamczak2008tail,Lederer2014,Sara2013Orliz,wellner2017bennett}.

Note that in finite time,  stationary points can be computed just approximately using gradient-based algorithms.
Now we  extend our results in Theorem~\ref{OracleInSta} to the points that are close but not necessarily equal to a stationary points.
We define a pair $(\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}})$ as a $\textcolor{\draftcolor}{\tau}-$approximate stationary point if it satisfies
\begin{equation}\label{approx-sta}
    \bigl|\objectiveF{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}+\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\dbtilde{\ParamV}}}-\objectiveF{\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}}-\textcolor{\draftcolor}{\ensuremath{r}}\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}\bigr|~\le~\textcolor{\draftcolor}{\tau}
\end{equation}
for a $\textcolor{\draftcolor}{\tau}\in[0,\infty)$.
Employing gradient based algorithms (in finite time), we can expect to get close to a stationary point in the sense that  $\textcolor{\draftcolor}{\dbtilde{\ParamV}}\approx\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}$~\citep{lei2019stochastic}.
Then also $\normone{\textcolor{\draftcolor}{\dbtilde{\ParamV}}}\approx\normone{\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}$, which means that
 an approximation of a reasonable stationary point is also reasonable once $\textcolor{\draftcolor}{\tau}$ is small enough.
Then, we extract statistical guarantees  for every practical network indexed by an approximate-reasonable stationary   as following:
\begin{theorem}[Statistical Guarantees for  Approximate Stationary Points]\label{StaAppr}
Suppose that $(\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}})$ is a $\textcolor{\draftcolor}{\tau}-$approximate stationary point and that the conditions of Theorem~\ref{OracleInSta} are satisfied.
Then, we have
\begin{equation}\label{main_ineq_app}
    \objectiveFP{\textcolor{\draftcolor}{\dbtilde{\parameterout}},\textcolor{\draftcolor}{\dbtilde{\parameterin}}}~\le~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+8\textcolor{\draftcolor}{\ensuremath{r}}\sqrt{\log\textcolor{\draftcolor}{\ensuremath{n}}}+\textcolor{\draftcolor}{\tau}
\end{equation}
with probability at least $1-1/\textcolor{\draftcolor}{\ensuremath{n}}$.
If $\textcolor{\draftcolor}{\ensuremath{r}}=\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}$,
the bound becomes
\begin{equation}
 \ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin}]~\leq~\ensuremath{\textcolor{\draftcolor}{\ensuremath{ \operatorname{risk}}}}[\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*},\textcolor{\draftcolor}{\ensuremath{\Theta^*}}]+\textcolor{\draftcolor}{\nu}(\log{\textcolor{\draftcolor}{\ensuremath{n}}})^{2}\sqrt{\frac{\log(\textcolor{\draftcolor}{\ensuremath{n}}\textcolor{\draftcolor}{p})}{\textcolor{\draftcolor}{\ensuremath{n}}}}+\textcolor{\draftcolor}{\tau}\,.
\end{equation}

\end{theorem}
The proof of Theorem~\ref{StaAppr} is given in Section~\ref{proofs}.

The bounds match the earlier ones with only two small differences: 1.~a summand $\textcolor{\draftcolor}{\tau}$ is added to our statistical bounds and 2.~the factor 5 in Equation~\eqref{main_ineq} is replaced by a factor of 8 in Equation~\eqref{main_ineq_app}.
Theorem~\ref{StaAppr} might look like a trivial extension of Theorem~\ref{OracleInSta},
but seem to be a trivial observation,  but the fact that Equation~\eqref{approx-sta} involves the (in-sample-)prediction risk and the   sparsity factors make the proof considerably more involved.

\section{Numerical observations}\label{NumSec}
 We provide here some numerical observations to clarify  theories  of Section~\ref{Mainresult}.
 We minimize a  least-squares complemented by $\ell_1$-regularization for shallow neural networks with 1.~linear and 2.~ReLU activation functions.
We consider
neural networks with $\textcolor{\draftcolor}{\ensuremath{d}}=\textcolor{\draftcolor}{\ensuremath{w}}=10$, that are trained over  500  and tested over 300 data sample generated from a  standard normal distribution and labeled by a sparse-target network (having the same structure as above) plus a Gaussian noise.
Note that here, we train the networks in a finite time, that means,  trained networks are just  an approximation of a stationary point (due to the non-convexity).
We report the relative training error and the relative test error for a potential global optima   and an approximate stationary point  for linear and ReLU networks in Table~\ref{Tab-optim}.
Let note that  the potential global optima and approximate stationary point are reached over multiple times of training on a fixed data set and assigned by the  trained networks with the lowest and  highest training error, respectively (we reference to Figure~\ref{fig:Optim_Linear&Relu} in Section~\ref{proofs} for a graphical view of  convergence in training).
Due to the non-convexity of the objective function, global optimization is far reaching and so we call the trained network with the lowest training error just as a
potential global optima.
 Results reveal that the test error for a potential global optima and an approximate stationary point are very close in both linear and ReLU networks.

These observations reveal that: First, global optimization for neural networks is far reaching even for very simple neural networks. Second, very practical outputs in deep learning (approximate stationary points) can still generalize  well---for linear networks and beyond.

 \begin{center}
\begin{table}
\begin{tabular}{|c|cc|cc|}
\hline
                             & \multicolumn{2}{c|}{Linear}                      & \multicolumn{2}{c|}{ReLU}                        \\ \hline
                             & \multicolumn{1}{c|}{Training Error} & Test Error & \multicolumn{1}{c|}{Training Error} & Test Error \\ \hline
Potential Global Optima      & \multicolumn{1}{c|}{1}              & 1          & \multicolumn{1}{c|}{1}              & 1          \\ \hline
Approximate Stationery Point & \multicolumn{1}{c|}{1.15}           & 1.0001     & \multicolumn{1}{c|}{1.11}           & 1.09       \\ \hline
\end{tabular}\caption{relative training error  and  test error for trained  neural networks (with $\textcolor{\draftcolor}{\ensuremath{d}}=\textcolor{\draftcolor}{\ensuremath{w}}=10$)  with   linear and ReLU  activations in a potential global optima and an approximate stationary point}
\label{Tab-optim}
\end{table}
\end{center}

\section{Technical results}\label{TechResult}
In this section, we provide technical results that  are essential for proving our Theorem~\ref{OracleInSta} but might also
 be of interest by themselves.
\paragraph{Additional notation}\label{AddNotations}
For  vectors
$\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p}}$ and $\boldsymbol{\textcolor{\draftcolor}{\alpha}}\deq(\textcolor{\draftcolor}{\alpha}_1,\dots,\textcolor{\draftcolor}{\alpha}_{\textcolor{\draftcolor}{\ensuremath{w}}})\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$  with $\textcolor{\draftcolor}{\alpha}_{\textcolor{\draftcolor}{\ensuremath{j}}}\ne 0$ for all $\textcolor{\draftcolor}{\ensuremath{j}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$, we
define $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}}\deq\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}})\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p}}$ as a rescaled version of $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}$ with $(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}})_{\textcolor{\draftcolor}{\ensuremath{j}}}\deq\textcolor{\draftcolor}{\parameteroutE_j}\cdot\textcolor{\draftcolor}{\alphaN_{j}}$ and  $(\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}})_{\textcolor{\draftcolor}{\ensuremath{j}}\textcolor{\draftcolor}{\ensuremath{k}}}\deq\textcolor{\draftcolor}{\ensuremath{\parameterinE_{jk}}}/\textcolor{\draftcolor}{\alphaN_{j}}$ for all $\textcolor{\draftcolor}{\ensuremath{j}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{w}}\}$ and $\textcolor{\draftcolor}{\ensuremath{k}}\in\{1,\dots,\textcolor{\draftcolor}{\ensuremath{d}}\}$.
  We tabulate the second order partial derivatives of $\objectiveFP{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}$ with respect to the $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}=\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})$ in a matrix called $\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}}}\in \textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p}\times\textcolor{\draftcolor}{p}}$.
We use $\textcolor{\draftcolor}{{e}_{\operatorname{min}}}[\cdot]$ to generate the smallest eigenvalue of a matrix.
We use the notation $\boldsymbol{0}$ to generate a vector of zeros.

Now, we study the behavior of the Hessian matrix in a rescaled network as following:

\begin{proposition}[Hessian Behavior]\label{PSDHessian}

Suppose that $(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}},\textcolor{\draftcolor}{\ensuremath{\Theta}})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}$ with
 $\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top$ invertible.
Let $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}\deq[(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}})^\top,(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}})^\top]^\top\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{p}}$ be a vector with $\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}}=1$, $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}\cdot\textcolor{\draftcolor}{\ensuremath{d}}}$, and $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$.
If $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}= \boldsymbol{0}$ or $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}= \boldsymbol{0}$,
we have for all $\boldsymbol{\textcolor{\draftcolor}{\alpha}}\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}\setminus\{\boldsymbol{0}\}$
\begin{equation*}
    \textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}^\top\HessianF{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}}}\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}}}~\ge~0\,;
\end{equation*}
Otherwise, above inequality holds for
all $\boldsymbol{\textcolor{\draftcolor}{\alpha}}\deq(1/\textcolor{\draftcolor}{{c}},\dots,1/\textcolor{\draftcolor}{{c}})\in\textcolor{\draftcolor}{\mathbb{R}}^{\textcolor{\draftcolor}{\ensuremath{w}}}$ with $\textcolor{\draftcolor}{{c}}\in[1,\infty)$ such that
\begin{equation*}
    \textcolor{\draftcolor}{{c}}^2 \ge \frac{2\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}}^2\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}^2 +4\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}}\normtwoB{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}\normtwoB{\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}^\top\textcolor{\draftcolor}{\ensuremath{\Theta}}-\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}^*}^\top\textcolor{\draftcolor}{\ensuremath{\Theta^*}}}}{\textcolor{\draftcolor}{{e}_{\operatorname{min}}}\bigl[\textcolor{\draftcolor}{\ensuremath{\Theta}}\parameterin^\top\bigr]\normtwo{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}}^2}\,.
\end{equation*}
\end{proposition}
The proof of Proposition~\ref{PSDHessian} is given in Section~\ref{proofs}.

Note  that if $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^1}}= \boldsymbol{0}$ or $\textcolor{\draftcolor}{\ensuremath{\boldsymbol{a}^2}}= \boldsymbol{0}$, the quadratic product on the Hessian matrix (in a rescaled network with parameters $(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\textcolor{\draftcolor}{\boldsymbol{\alpha}}})$) is non-negative for all $\textcolor{\draftcolor}{\boldsymbol{\alpha}}$, otherwise, it is  non-negative just for $\textcolor{\draftcolor}{\boldsymbol{\alpha}}$ with large enough $\textcolor{\draftcolor}{{c}}$.

\iffalse

Result above  shows the behavior of Hessian matrix (population risk) in a scaled pair $(\textcolor{\draftcolor}{\boldsymbol{\parameteroutE}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}},\textcolor{\draftcolor}{\ensuremath{\Theta}}_{\boldsymbol{\textcolor{\draftcolor}{\alpha}}})$ and reveals that for large enough $\textcolor{\draftcolor}{{c}}$, which is the scaling parameter, the Hessian matrix is  ``approximately positive semidefinite".

\fi
\begin{lemma}[Empirical Processes]\label{EmpProcctwo}
  It holds for each  reasonable stationary  point $\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}} =\textcolor{\draftcolor}{{\operatorname{vec}}}(\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin})$ of the objection function in~\eqref{ls} that
\begin{align*}
     \Bigl|\bigl(\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}-\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}\bigr)^\top(\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}})\Bigr|
     &\le \textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}\normone{\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}^*}}-\textcolor{\draftcolor}{\widetilde\textcolor{\draftcolor}{\ensuremath{\boldsymbol{\beta}}}}}+\frac{\textcolor{\draftcolor}{\ensuremath{\tuning_{\operatorname{orc}}}}}{2\textcolor{\draftcolor}{\ensuremath{n}}}
\end{align*}
with probability at least $1-1/{2\textcolor{\draftcolor}{\ensuremath{n}}}$.
\end{lemma}
The proof of Lemma~\ref{EmpProcctwo} is given in Section~\ref{proofs}.

The result above establishes  a bound for the absolute difference between $\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveF{\StaOut,\StaIn}}}}$ and $\textcolor{\draftcolor}{\ensuremath{\nabla{\objectiveFP{\StaOut,\StaIn}}}}$ for every reasonable stationary point  $(\textcolor{\draftcolor}{\widetilde\parameterout},\textcolor{\draftcolor}{\widetilde\parameterin})\in\textcolor{\draftcolor}{\ensuremath{\mathcal{B}}}$.  We employ the result above for choosing the optimal tuning parameter. Proposition~\ref{PSDHessian} and Lemma~\ref{EmpProcctwo} are the main ingredients for proving our Theorem~\ref{OracleInSta}.

\section{Related literature}\label{RelatedWork}
\input{Sections/Relatedworks}
\section{More technical results and proofs}\label{MoreResult}
In this section, we establish five auxiliary results and then the proofs of all our claims.
\subsection{More technical results}\label{MoreTechResult}
\input{Sections/technicalresults}
\subsection{Proofs}\label{proofs}
\input{Sections/proofs}
\section{Discussion}\label{discussion}
This paper establishes statistical guarantees for approximate stationary points of regularized  neural networks.
Despite being limited to linear networks,
the theory is a considerable step forward in three ways:
First, previous empirical studies show that the optimization theory of linear-neural networks may exhibit some properties of non-linear models as well~\citep{saxe2013dynamics},
which means that our theories should also give some insights into non-linear networks (see also our Section~\ref{NumSec}).
Second, several papers consider the existence or non-existence of spurious local minima,
but importantly, our theory also applies to saddle points,
which can be also hard to escape from in optimization.
And third,
our new statistical approach inspired by high-dimensional statistics will hopefully spark further progress in the mathematical understanding of deep learning.

\paragraph{Acknowledgements}
M.T.~and J.L.~acknowledge funding from the German Research Council through the Research-Training Group 2131.