\section{Introduction}
The many--body localization (MBL)\cite{gornyi05,basko_aleiner_06,oganesyan_huse_07,AbaninRMP} phenomena has been most frequently studied in one--dimensional (1D) disordered systems with either  charge or spin degrees of freedom \cite{barisic_prelovsek_10,luitz15,luitz16,torres15,sirker14,pal10,bera15,Hauschild_2016,Devakul2015,bertrand_garciagarcia_16,Husex2017,Doggen2018}.
Even though    research in this field  mainly focused on a few simplest prototype model Hamiltonians for MBL such as the disordered  XXZ model, the type of the transition and even the existence  of the MBL phase in the thermodynamic limit are  still under intense consideration\cite{suntajs_bonca_20a,suntajs2020, sirker_2020, Luitz20, Vasseur20,Polkovnikov20,sieran2021a,vidmar2021,Huse21,sels2021,Sirker21}. One of the specific open problems is the effect of various  symmetries on the existence of MBL phase\cite{Vasseur2016,Chandran2014}. There are reports that non--Abelian symmetry precludes MBL \cite{Vasseur2016,proto2017,vasseur2021} while other investigations  claim that the non-Abelian symmetry
is protected by  MBL\cite{friedman2017}.

Shifting focus  to more complex prototype models that contain charge and spin degrees of freedom, such as the 1D  Hubbard model with potential disorder, the existence of the full MBL phase is even more elusive. In Ref.~\cite{prelovsek16,sroda19,kozarzewski2019,protopopov2018} authors investigate  the time evolution of spin and charge imbalance as well as transport properties  in the Hubbard model. Their results are consistent with localized/nonergodic  charge degrees of freedom while due to the preserved SU(2) symmetry the spin degrees of freedom remain  delocalized/ergodic up to extremely large values of potential disorder.  Similar conclusions were drawn based on the statistics of adjacent energy levels \cite{mondaini15} and by counting  the maximal number of local integrals of motion\cite{mierzejewski2018}. Subdifusive time evolution of charge particles was found   in the related   $t-J$ model\cite{bonca17} with potential disorder.

The effect of symmetry preserving disorder has been addressed already in non--interacting one-dimensional random hopping systems. In the case of systems with chiral or sublattice symmetry where particles can hop only between even- and odd- lattice sites, there is a  diverging mean density of states at zero--energy  \cite{dyson1953,cohen1976,reidinger1978},  which can lead to the delocalization transition \cite{brouwer1998,furusaki_NPB,furusaki2000,evers2008}.

The main goal of this work is to compare  the effect of spin-symmetry-preserving and symmetry-breaking disorders  on the dynamics of charge and spin degrees of freedom. In the case of  potential disorder, the behavior of specific  spin degrees of freedom
 is inconsistent with  the full MBL state due to the   $\mathbb{Z}_2$ spin-symmetry.  This observation is based on relatively fast relaxation of spin observables
 that are odd under the later  $\mathbb{Z}_2$  spin transformation, i.e.,
the only non--zero matrix elements of these operators connect two distinct  symmetry sectors. While this observation    seems to preclude MBL state,
  the  statistics of  adjacent energy  levels computed within each symmetry sector at large potential disorder approaches Poisson statistics.

We have organized the manuscript as follows: in the introduction, we present the model and the numerical method; we also discuss how the symmetry properties of the model depend on the type of the disorder. Next, we present the time evolution of the charge and spin imbalance and present a simplified model that explains the unusual relaxation of the spin imbalance in the presence of the potential disorder. We further present time evolutions of various charge and spin correlation functions, followed by the analysis of  charge and spin entanglement entropies.
Based on the spectral level statistics, we discuss the apparent inconsistency between the relaxation of the spin imbalance and the Poisson level statistics, both observed under the influence of strong potential disorder.

\section{Model and method.}
We investigate the
anisotropic $t-J$ model  on a one--dimensional ring with $L$--sites and $N_f=L/2$ fermions in the total $S^z=0$ subspace in the presence of either   a random external magnetic field $h_i \in [-2W_s,2W_s]$ or random potential $\epsilon_i \in [-W_c,W_c]$
\begin{eqnarray}
H &=& -t_0\sum_{i,\sigma}  c^{\dagger}_{i,\sigma} c_{i+1,\sigma} + c.c. +  J_z \sum_{i}S^z_iS^z_{i+1}- n_i  n_{i+1}/4  \nonumber \\ &+& {J_\perp \over 2}\sum_i  S^+_i S^-_{i+1} + S^-_i S^+_{i+1} + \sum_i h_iS^z_i + \epsilon_i  n_i.
\label{ham}
\end{eqnarray}
The fermion  operators,  $ c_{i,\sigma} $ and $ n_i = \sum_\sigma  c_{i,\sigma}^\dagger  c_{i,\sigma}$, as well as  the spin operators, $S^{\pm,z}_i$,
act in the Hilbert space spanned locally by only three states, $|0\rangle_i$, $|\uparrow \rangle_i$, $|\downarrow \rangle_i$. Absence of doubly-occupied states, $|\uparrow \downarrow  \rangle_i$, in the $t$-$J$ model allows studying
charge and spin dynamics for much  larger systems than it would be possible for the Hubbard chain. This property is  the main motivation for the choice of Hamiltonian.
We perform multiple  time-evolutions based on the  Lanczos  technique,   each evolution starting from a different set of either random ${h_i}$  or  ${\epsilon_i}$.  The main goal of this work is to compare the time evolution of spin and charge degrees  of freedom under the influence of two different types of disorder. For this reason we choose,  following Ref.\cite{mondaini15}, the  initial state that  possesses  a charge--density--wave order as well as a staggered  spin orientation configuration $\vert \Psi_0\rangle$ defined as
\begin{eqnarray}
\vert \Psi_0\rangle &=& \vert \uparrow 0 \downarrow 0 \uparrow 0 \downarrow 0 \dots \rangle,\\
\vert \bar\Psi_0\rangle &=& \hat{U} \vert \Psi_0\rangle = \vert \downarrow 0 \uparrow 0 \downarrow 0 \uparrow 0 \dots \rangle,
\end{eqnarray}
while $\vert \bar\Psi_0\rangle$ represents a state with a globally reversed spin projections, $S^z_i$ , via the unitary transformation  $\hat{U}=\prod_i (1-  n_i+S^+_i +S^-_{i})$.  In addition, we compute the level statistics of the  energy spectra.  We typically take $N_r=500-1500$ realisations of the disorder. We measure time in units of $[1/t_0]$ and set $t_0=J_z=1$, and $J_\perp=1.5$.

There exists a significant  difference  between the two systems under the influence of  potential ($W_c\not =0, W_s=0$) and field disorder ($W_s\not =0, W_c=0$). Since we have set $J_z\not = J_\perp$, the $SU(2)$ symmetry is broken even at $W_{c(s)}=0$. In the case of the potential  disorder and for   $S^z_\mathrm{tot}=0$,
 the Hamiltonian remains invariant under the $\mathbb{Z}_2$ spin-transformation, $\hat{U}$,
which is closely related to the  global $\pi$-rotation around the $x$--axis  \cite{santos_2013}.
Since $\hat{U} \hat{U} =\hat{U}  \hat{U}^{\dagger} =1$,   each eigenstate consists of either symmetric ($\hat{U}  \vert \Psi_{S} \rangle=  \vert \Psi_{S} \rangle$)  or antisymmetric ($\hat{U}  \vert \Psi_{A} \rangle= - \vert \Psi_{A} \rangle$) combination of states
that differ by a global reversal of $S^z_i$,

  $\vert \Psi_{S/A}\rangle = (\vert \Psi \rangle \pm \hat{U} \vert \Psi \rangle)/2$. The Hamiltonian thus  separates into two blocks with equal number of symmetric and antisymmetric functions.

\section{Charge and spin imbalance}
We start by presenting time propagation of charge and spin imbalance as defined by the following operators
\begin{eqnarray}
\hat P_c &=& {1\over N_f}\sum_{i=1}^L (-1)^{i+1} n_i \\
\hat P_s &=& {2\over N_f}\sum_{i=1}^L (-1)^{\sum_{j=1}^{i-1}  n_j}S^z_i
\end{eqnarray}
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{Fig1.pdf}
\caption{$P_{c(s)}(t)=\langle\hat P_{c(s)}\rangle_t$ for a system with $L=16$. Note that $P_{s(c)}(t=0)=1$. Results using finite potential disorder $W_c$ (and $W_s=0$) are presented with full lines while those with field  disorder $W_s$ (and $W_c=0$) with dotted lines. Roughly $N_r\sim 500$ realizations have been used for each set of data. The inset in (b) represent the relaxation time $\tau$ defined through intercepts of $P_s$ with 0 obtained by fitting $P_s(t)=A\log(t)+B$ in the interval $t\gtrsim 4$ and $t\sim \tau_\mathrm{approx}$. Fits are displayed by thin dashed red lines. A brown  dashed line in the inset represents  a single--parameter  fit $\tau=0.28 W_c^2$. Similar fits as for $P_s(t)$  in a different time interval for $P_c(t)$ are also displayed in Fig.~\ref{Fig1}(a) using thin dashed red lines. Relaxation times for $W_{c(s)}=14$ based on presented fits are $\tau_c = 4.7\times 10^6$ and $\tau_s=7.7 \times 10^{16}$.  Small time step $\Delta t=0.01$ have been used in all presented results to obtain sufficient  numerical stability of time propagation. c) Processes that connect two  spin--reversed states with equal energy. Here,  $H_{t_0}$ and $H_\perp$ represent the first and the third term in Eq.~\ref{ham} for potential disorder.  }
\label{Fig1}
\end{figure}
The  initial state is chosen such that $\hat P_{c(s)}\vert \Psi_0\rangle = \vert\Psi_0\rangle$.  Note also that $\hat P_s$ is odd under the $\mathbb{Z}_2$ spin-transformation,
$\hat U \hat P_s \hat U =-  \hat P_s$,  thus it connects the symmetric with the antisymmetric sectors, while its matrix elements within each symmetry block vanish,
$\langle \Psi_{S} \vert   \hat P_s \vert  \Phi_{S} \rangle=\langle \Psi_{A} \vert   \hat P_s \vert  \Phi_{A} \rangle=0$.
Consequently, in the basis of the eigenstates of the Hamiltonian,   $\hat P_s$ has no diagonal matrix-elements.
In contrast $\hat P_{c}$  is even, $\hat U \hat P_c \hat U =  \hat P_c$  and  $\langle \Psi_{S} \vert   \hat P_c \vert  \Phi_{A} \rangle=0$.

We first show the charge imbalance, presented in Fig.~\ref{Fig1}(a), $P_c(t) = \langle \hat P_{c} \rangle_t$, where  $\langle \dots \rangle_t$  indicates multiple time evolutions from the initial $\vert \psi_0\rangle$, averaged over different  random realizations of either potential or magnetic field disorder.
We observe  a similar time evolution for $t\lesssim 10$  under the influence of either potential  or field  disorder. At larger $t$ and for
$W_{c(s)} \leq 6$, $P_c(t)$ relaxes towards zero faster  in the case of field  disorder. At larger  $W_{c(s)} \geq 10$
 $P_c(t)$ shows slow, logarithmic decrease in time. Providing that the time dependence would further follow a logarithmic form $A\log(t)+B$, as displayed with  thin dashed lines for $W_{c(s)}=10$ in Fig.~\ref{Fig1}(a), $P_c(t)$ would equilibrate under $W_c$ or $W_s$ to zero around $\tau_c\sim 27.000$ and $\tau_s\sim 460.000$, respectively. It is worth stressing that relaxation time becomes longer in the case of potential disorder  at larger values of $W_{c(s)}\geq 10$, moreover, relaxation might be prevented by the onset of the many body localization.

   In the case of the spin imbalance $P_s(t)$, see Fig.~\ref{Fig1}(b), we find exceedingly  different time evolution when comparing  systems  with potential  or field  disorder. While $P_s(t)$ in the case of the field  disorder shows qualitatively similar behaviour as $P_c(t)$, the time evolution in the case of the potential disorder shows relaxation for all $W_s$ on a time scale accessible by our calculations. Moreover, the corresponding relaxation times $\tau$ show a quadratic $W_c$--dependence as shown in the inset of Fig.~\ref{Fig1}(b).  The latter quadratic dependence may be explained by recalling that for large $W_c, (W_s=0)$ a  redistribution of charge may be energetically very costly, however reversing the spin orientation does not change the energy.
Therefore the charge redistribution that is necessary for the spin dynamics is realized via virtual processes shown in  Fig.~\ref{Fig1}(c). In order to estimate the relevant energy scale (i.e. also the time-scale) one may study a toy model with four local states shown in  Fig.~\ref{Fig1}(c): two initial states $\vert \uparrow,0,\downarrow \rangle^{S/A}$ and two virtually-generated states $\vert \uparrow,\downarrow ,0\rangle^{S/A}$.  The corresponding eigenproblem  is up to a constant energy shift given by  $2 \times 2$ matrices
\begin{equation}
\left(
\begin{array}{cc}
0 & -t_0 \\
-t_0 & V^{S/A}
\end{array}
\right)
\left(
\begin{array}{c}
\vert \uparrow,0,\downarrow \rangle^{S/A}  \\
\vert \uparrow,\downarrow ,0\rangle^{S/A}\\
\end{array}
\right)
=E^{S/A}
\left(
\begin{array}{c}
\vert \uparrow,0,\downarrow \rangle^{S/A}  \\
\vert \uparrow,\downarrow ,0\rangle^{S/A}\\
\end{array}
\right),
\label{exple}
\end{equation}
where the potentials $V^{S/A}=\epsilon_2-\epsilon_3-\frac{1}{2}(J_{z} \mp J_{\perp}) $. If the charge disorder is strong then the typical values of  $\vert V^{S/A} \vert \sim \vert \epsilon_2-\epsilon_3 \vert$ are large. As a consequence only two eigenstates have large projections on the initial states, $\vert \uparrow,0,\downarrow \rangle^{S/A}$.
Then, the dynamics of odd  operators is governed by the difference of corresponding eigenenergies in symmetric and antisymmetric sectors \mbox{ $\tau=|E^S-E^A|^{-1}\sim W_c^2/ t_0^2J_\perp $.}

\begin{figure}
\centering
\includegraphics[width=\columnwidth]{Fig2.pdf}
\caption{  $G_n(t)$ in  (a) and $G_{S^z}(t)$ in (b) for different strengths of disorder. Full lines represent  potential disorder, $W_c$, while dotted lines magnetic field disorder, $W_s$. Thin horizontal red dashed lines represent infinite--$T$ limits of correlation functions, $G_n(T\to\infty)=7/30, G_{S^z}(T\to \infty)=-1/120$. Note, their respective values in the thermodynamic limit are $1/4$ and 0, respectively.  Parameters of the system are the same as in Fig.~\ref{Fig1}. }
\label{Fig2}
\end{figure}
\section{Charge and spin correlation functions}
 We next explore the neighboring  density--density and spin--spin correlation function defined as:
\begin{eqnarray}
\hat G_n = {1\over L}\sum_i   n_i  n_{i+1}, \quad \quad G_n(t)=  \langle  \hat G_n \rangle_t,\\
\hat G_{S^z}={1\over L}\sum_i  S^z_i S^z_{i+1},\quad \quad G_{S^z}(t)=\langle   \hat G_{S^z}\rangle_t,
\end{eqnarray}
For the proper analysis of long--time behavior it is important to note that the  energy of the initial state after averaging over different random realizations, $E_\mathrm{ave}=\langle \psi_0\vert H\vert\psi_0\rangle_\mathrm{ave}=0$, is located near the middle of the energy spectrum, which in the microcanonical sense corresponds to infinite--temperature ($T\to \infty$).
Based on the eigenstate thermalization hypothesis  \cite{srednicki_94, rigol_dunjko_08, deutsch_91, dalessio_kafri_16, eisert_friesdorf_15, deutsch_18, mori_ikeda_18} it is expected  that for small $W_{c(s)}$  and $t\to \infty$ $G_{n(S^z)}$ approach   their respective $T\to \infty$ limits, as indicated by dashed horizontal lines in Fig.~\ref{Fig2}.  It is indicative that charge and spin correlation functions for short times, $t\lesssim 1$ display qualitatively similar time dependence. Initially, only the hopping part  (the first term) of the Hamiltonian in Eq.~(\ref{ham}) is active since the exchange interaction can only act between  particles  on neighbouring sites. The change of either potential energy or field energy after hopping between neighboring lattice sites is in both cases comparable, which leads to a similar time--dependence for times comparable to inverse hopping time $1/t_0$.  For larger $W_{c(s)}\gtrsim 10$ and $t\gtrsim 50$,  $G_n$ shows logarithmic increase in time while $G_{S^z}$ shows logarithmic decrease. The difference is due to substantially different $T\to\infty$ limits. In contrast to $G_n$, $G_{S^z}$ shows  distinct dependence with regard to the  type of disorder. At large $W_c \geq10$,  $G_{S^z}$ approaches significantly  closer the ergodic $T\to \infty$ limit than in the case of $W_s \geq10$. Still, $G_{S^z}$ does not show relaxation as is the case of $P_s(t)$, shown in Fig.~\ref{Fig1}(b). The explanation for this seeming discrepancy can be found again in the symmetry argument. While $\hat P_s$ is odd under
the  $\mathbb{Z}_2$ spin-transformation,  $\hat G_{S^z}$ is even, it has non-zero matrix elements within a fixed  symmetry sector, and the matrix elements  which are diagonal in the eigenstates of Hamiltonian may be non-zero as well.

To test this assumption,  we define a  three--site operator
\begin{equation}
\hat \Gamma = {8\over N_f}\sum_{i=1,3,5,\dots,L-1}(-1)^{1+(i+1)/2} S^z_iS^z_{i+2}S^z_{i+4},
\end{equation}
which is also odd under the  $\mathbb{Z}_2$ spin-transformation, \mbox{$\hat U \hat \Gamma \hat U = - \hat \Gamma$}.
As seen in Fig.~\ref{Fig3}, $\Gamma(t) =\langle \hat \Gamma \rangle_t$ as well shows relaxation with the respective relaxation times $\tau_\Gamma$ scaling with $W_c^2$ just as in the case of $P_c(t)$, shown in Fig.~\ref{Fig1}(b).
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{Fig3.pdf}
\caption{  $\Gamma(t)=\langle\hat\Gamma\rangle_t$  for different strengths of disorder. Full lines represent  potential disorder $W_c$ while dotted lines magnetic field disorder $W_s$. Dashed red lines represent  fits of the form $\Gamma(t)=A \ln(t)+B$ that were used to extract relaxation times $\tau_\Gamma$, presented in the inset. Dashed brown line in the inset represents a single--parameter fit $\tau_\Gamma=0.45W_c^2$.  Parameters of the system are the same as in Fig.~\ref{Fig1}. }
\label{Fig3}
\end{figure}

\section{Spin and charge entanglement entropy}

We now turn to  comparison  of the dynamics  of the entanglement
properties of charge and spin degrees of freedom. To this end we split the system into two halves. We then compute the charge and spin entanglement entropy\cite{lukin_rispoli_19,sirker_2020}, respectively:
\begin{eqnarray}
S_n &=& -\sum_{n=0}^{N_f} p_n \log(p_n)\\
S_{S^z}&=& -\sum_{S^z=-S^z_\mathrm{max}}^{S^z_\mathrm{max}} p_{S^z} \log(p_{S^z}),
\end{eqnarray}
where $S^z_\mathrm{max}=N_f/4$ and $p_n$ and $p_{S^z}$ represent probabilities that the subsystem  contains either   $n-$ fermions or  the $z-$component of the total spin equals $S^z$, respectively.

The charge entropy $S_n$, shown in Fig.~\ref{Fig4}(a), displays  qualitatively similar behavior with respect to either the potential or  the  field  disorder for small $W_{c(s)}\leq 6$. In both cases $S_n$ approaches its maximal value, characteristic for a thermal state at  $t\to \infty$. For larger $W_{c(s)}\geq 10$  we observe a stronger deviation for different types of disorder in the long--time limit. In both case se observe  a slow  logarithmic increase, characteristic for  MBL systems.%
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{Fig4.pdf}
\caption{$S_n(t)$ in a) and $S_{S^z}(t)$ in b) for different strengths of disorder. Full lines represent  potential disorder $W_c$ while dotted lines magnetic field disorder $W_s$. Results are normalized to their respective  values in the infinite--$T$ limit, $S_{n}^\mathrm{max}\sim 1.45$ and $S_{S^z}^\mathrm{max}\sim 1.80$.  Parameters of the system are the same as in Fig.~1.  }
\label{Fig4}
\end{figure}

In contrast, the spin entropy $S_{S^z}$ quantitatively differs in comparison to potential  or field  disorder in the whole range of $W_{c(s)}$'s. The most significant difference is observed when comparing results for larger $W_{c(s)}\geq 10$ where $S_{S^z}(W_c)$ grows significantly faster than $S_{S^z}(W_s)$. For example: while at $W_{c(s)}=10$ $S_{S^z}(W_c)$ nearly reaches its maximal value $S_{S^z}^\mathrm{max}$, $S_{S^z}(W_s)$ displays slow logarithmic growth.  For even larger $W_{c}\geq 14$ there seems to be no observable time interval with a logarithmic growth of $S_{S^z}(W_c)$. In contrast, it shows a tendency towards saturation towards a non--thermal value.

  To gain a deeper physical picture  we first note that while the field  disorder affects charge as well as spin degrees of freedom, the potential  disorder affects only charge degrees of freedom. For example: when a fermion with spin--up hops between neighboring sites under the influence of field disorder, it feels different Zeeman energy just as when in the presence of the potential  disorder it feels different potential energy.  There exist connected spin chains separated by empty sites. Spins that form a particular connected spin chain do not feel the potential  disorder as long as they remain attached to the chain. Spin excitation  can freely propagate along a connected spin chain in the presence of the potential  disorder. When such connected spin chains extend between the boundaries of the two subsystems, they contribute to the growth of spin entropy.

\section{Spectral level statistics}

Motivated by the rather unexpected difference in the time evolution of charge vs. spin imbalance, observed in Fig.~\ref{Fig1}(b) as well as other observables  probing  charge or spin degrees of freedom  under the influence of potential  disorder, we next explore statistical properties of the energy spectra. We compute adjacent energy level   spacing ratios, $r_n =\mathrm{min}[\Delta_n,\Delta_{n-1}]/\mathrm{max}[\Delta_n,\Delta_{n-1}]$ where $\Delta_n = E_n - E_{n-1}$ and $\{E_n\}$ represents the ordered set of energy levels of Hamiltonian in Eq.~(\ref{ham}).  For each realization of disorder we compute average value of $\bar r$ and then instead of computing average over different realizations, we calculate  the cumulative distribution function for $\bar r$, $F(\bar r)$. In Fig.~\ref{Fig5}(a) we first present $F(\bar r)$ for the case of potential disorder, $W_c$, taking into account the full spectrum. Since the Hamiltonian in Eq.~(\ref{ham}) is non--integrable, one expects that at small $W_c=2$ its spectrum resembles the spectrum of the Gaussian random matrices where $\bar r_\mathrm{ave} = \bar r_\mathrm{GOE} \simeq 0.53 $\cite{mukerjee_2006,oganesyan_huse_07}.  In contrast, at large $W_c = 20$, the average value of $\bar r$ should not drop below $r_\mathrm{Poisson}=0.386$, characteristic for a random distribution of energy levels. Distributions $F(\bar r)$, shown in Fig.~\ref{Fig5}(a) are not consistent with either  of the above predictions.

For a proper analysis of the spectral level statistics in the case of the potential disorder we have computed $F(\bar r)$ separately for each symmetry subspace. In Figs.~\ref{Fig5}(b) and \ref{Fig5}(c)   we present $F(\bar r)$ for different values of $W_{c(s)}$. There are two types of nearly overlapping curves (full and dashed lines) in the case of potential disorder, see Fig.~\ref{Fig5}(b) representing separately $F(\bar r)$  for each symmetry sector. For small $W_{c(s)}=2$, presented in Figs.~\ref{Fig5}(b) and (c), $F(\bar r)$ can be fitted with the Error  function positioned  at $\bar r_\mathrm{ave}=0.53$,  which agrees with $r_\mathrm{GOE}$. At large $W_{c(s)}=20$ $F(\bar r)$ again resemble  Error functions, however in this regime  close to    $r_\mathrm{Poisson}$, which on a finite system indicate localization. For the intermediate  values of $W_{c(s)}\in [4,6]$ we observe broad distributions $F(\bar r)$ that  result from a mixture of systems where a part of them exhibit level statistics that resembles ergodic systems  while others the one closer to  non--ergodic/localized.
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{Fig5.pdf}
\caption{ $F(\bar r)$ for different values of potential  disorder in (a) from the full spectrum, (b) from the symmetric (full lines) and the antisymmetric (dashed lines) part of the spectra and for field  disorder in (c). Vertical dashes lines indicate values of
$r_\mathrm{Poisson} \simeq 0.386$ and $r_\mathrm{GOE} \simeq 0.53$. In most cases  $N_r\sim 1500$ samples have been used. Results were obtain using full diagonalization on a system with  $L=12$, $N_\uparrow=N_\downarrow = 3$ and $N_\mathrm{states}=18480$.}
\label{Fig5}
\end{figure}

We have also calculated the  distribution of the gap ratios without averaging $r_n$ over multiple energy levels. To this end we have generated a set containing $r_n$ for various
 $n$ as well as for various disorder realizations and calculated the probability density, $P(r)$, from the histogram of $\{ r_n\}$. While the distribution $F(\bar{r})$ in Fig. \ref{Fig5} contains information about differences between various  realizations
of disorder, such information is not directly encoded in $P(r)$. Nevertheless,  $P(r)$ allows for a more detailed comparison with the random matrix theory.
 For the  Poisson level statistics one gets \cite{oganesyan_huse_07}
\begin{equation}
P_{P}(r)=\frac{2}{(1+r)^2},
\label{pp}
\end{equation}
 while an approximate  distribution for the Gaussian-Orthogonal- Ensemble (GOE) can be derived from the  Wigner surmise \cite{atas2013,alet2022,fremling2022} for 3 energy levels
 \begin{equation}
P_{GOE}(r)=\frac {27}{4} \frac{r+ r^2}{(1 + r + r^2)^{5/2}}.
\label{pgoe}
\end{equation}

\begin{figure}
\centering
\includegraphics[width=\columnwidth]{Fig6.pdf}
\caption{Points show the probability density of the gap ratio  for charge disorder obtained from symmetric (S) or antisymmetric (A) parts of spectra.
The fractions in the legend mark the parts of spectra which were used for generating the distribution. The dashed  lines in (a) and (c) show Eqs. (\ref{pgoe}) and (\ref{pp}), respectively.
The dashed line in (b) shows distribution given by Eq. (20)  in Ref. \cite{fremling2022} with a normalization that is relevant for $r \in (0,1)$. }
\label{Fig6}
\end{figure}
\begin{figure}
\centering
\includegraphics[width=\columnwidth]{Fig7.pdf}
\caption{  The same as in Fig. \ref{Fig6} but for a full spectra containing symmetric and antisymmetric levels.
The dashed line in (a) shows distribution given by Eq. (24)  in Ref. \cite{fremling2022} with a normalization that is relevant for $r \in (0,1)$, whereas the dashed line in (b) shows Eq. (\ref{pp}).
 }
\label{Fig7}
\end{figure}

Figure \ref{Fig6} shows the distributions, $P(r)$, obtained from  the symmetric (S) or the antisymmetric (A) parts of spectra with charge disorder.
In order to identify artifacts arising from the presence of the localization edge,
the distributions have been  generated either from  all levels or only from a central part ($1/3$ or $1/9$) of the levels in the middle of the spectrum. Results obtained for all three cases accurately overlap
(see Figs.  \ref{Fig6}  and  \ref{Fig7} ) indicating absence of artifacts originating from the localization edge.

As expected, numerical results for weak disorder [Fig. \ref{Fig6}(a)]  accurately reproduce  Eq. (\ref{pgoe})
whereas for strong disorder shown in  Fig. \ref{Fig6}(c),  the distribution agrees with Eq. (\ref{pp}). In the vicinity of the transition, $P(r)$ can be well approximated by a mixed Wigner surmise discussed  very recently in Ref. \cite{fremling2022}.
More precisely, the dashed curve in  Fig. \ref{Fig6}(b) shows  distribution for $2 \times 2$ GOE matrices mixed with two uncorrelated energy levels, see Eq. (20)  in Ref. \cite{fremling2022}.
Such mixture of GOE and Poisson distributions may be interpreted as a coexistence  of  localized (insulating) and delocalized (metalic) domains whereby the absence of level crossings, $F(r \to 0) =0$ in Fig. \ref{Fig6}(b), indicates that localization within the former domains is not perfect. Similar results concerning the spatial separation of conducting and insulating domains have been recently found
 for the random-field Heisenberg model \cite{herbrych2022}.

Figure \ref{Fig7} shows the distribution of the gap ratios obtain for charge disorder from the full spectrum that includes both  symmetric and antisymmetric levels. Results obtained for weak disorder can be very accurately approximated  by two independent GOE ensembles. In particular,  dashed line in  Fig. \ref{Fig7}(a) shows the distribution derived for mixture of two  $2 \times 2$ GOE matrices, see Eq. (24)  in Ref. \cite{fremling2022}. Rather unexpected is the case of strong disorder when  $P(r)$  for $r \ll 1$ significantly exceeds the distribution derived for the Poisson statics, as it is shown in Fig. \ref{Fig7}(b).
Comparing Fig. \ref{Fig6}(c) and  \ref{Fig7}(b) one identifies an attraction between  symmetric and antisymmetric energy levels.
Such scenario is consistent with results for the average ratio shown in Fig. \ref{Fig5}(a).

\section{ Summary}
We have studied spin- and charge-dynamics in a disordered chain such that an unperturbed system has $\mathbb{Z}_2$ spin-symmetry. Then we considered two types of disorder: a random magnetic field that breaks the $\mathbb{Z}_2$ symmetry and
random charge potential, which preserves the spin-symmetry. In the former case with broken spin-symmetry, the dynamics of all studied observable are consistent with localization on finite lattices in that their expectation values do not approach the results for thermal equilibrium.
 However, for the symmetry-preserving disorder, the
observables that are odd under the $\mathbb{Z}_2$ spin transformation seem to thermalize while even observables do not. Numerical studies of the time-evolution for the symmetry-preserving disorder have been accompanied by the level statistics. Interestingly, the level statistics
obtained separately for odd and even symmetry sectors accurately reproduce a crossover/transition from the GOE for weak charge disorder to the Poisson distribution for the strong disorder. The time evolution and the level statistics suggest that localization exists within each symmetry sector, i.e. for odd or even eigenstates and observables with matrix elements only within a given symmetry sector. The apparent relaxation of odd operators is not inconsistent with the level statistics since such operators evolve between the two sectors. Similar numerical results have been found for the dynamics in the Hubbard model with charge disorder, which, however, has SU(2) symmetry\cite{prelovsek16,kozarzewski18,sroda19,kozarzewski2019,protopopov2019}. In the latter model, the spin imbalance decays subdiffusively \cite{prelovsek16,kozarzewski18,sroda19} while the spin-energy-density seems not to thermalize \cite{kozarzewski2019,protopopov2019}.
In our studies, we have not considered the stability of the localized phase in the thermodynamic limit, so we do not exclude that localization represents extremely slow dynamics that eventually may lead to a thermal equilibrium of an infinite chain.

 Comparison of the entanglement entropies of spin and charge degrees of freedom reveals an essential difference between the field disorder that affects charge and spin degrees of freedom and the potential disorder that influences only charge degrees of freedom.  Spins that form a particular connected spin chain do not feel potential disorder. Spin excitations can thus freely propagate along a connected spin chain. This may explain the absence of a logarithmic growth of the spin entanglement entropy even in the regime of strong potential disorder, where in contrast, the charge entanglement entropy displays logarithmic time evolution.

The original motivation for this work stems from experiments on cold atoms \cite{schreiber_hodgman_15} where charge imbalance was measured in cold atoms experiment setup. Recent advances in the spin- and density--resolved microscopy\cite{Bloch2016,Bloch2020,Bloch2020a} might allow measurements of the charge and the spin imbalance under the non--symmetry--breaking disorder.

\acknowledgements
We acknowledge the support by the National Science Centre, Poland via project 2020/37/B/ST3/00020 (M.M.), the support by the Slovenian Research Agency (ARRS), Research Core Fundings Grants P1-0044 ( J.B.) and the support from  the Center for Integrated Nanotechnologies, an Office of Science User Facility operated for the U.S. Department of Energy (DOE) Office of Science by Los Alamos National Laboratory (Contract 89233218CNA000001) and Sandia National Laboratories (Contract DE-NA-0003525) (J.B.).
\bibliographystyle{biblev1}


\section{Introduction}
\label{sec:intro}

Monocular 3D human pose estimation aims to accurately regress the 3D locations of human joints in the camera coordinate system from a single image. It plays an important role in many applications, such as action recognition and human-computer interaction. Compared to the monocular systems, multi-view capture systems are expensive and inconvenient to set up and operate, which prevents them from being widely used in practice.
To tackle the monocular 3D HPE task, some approaches~\cite{cai2019exploiting, lee2018propagating, sun2017compositional, sun2018integral, yang20183d} regress 3D joint coordinates or heat maps directly from an image via a convolutional neural network (CNN)~\cite{krizhevsky2017imagenet, lecun1998gradient}. Directly regressing from the image space suffers from the large parameter searching space, which always leads to a sub-optimal solution.
Recently, Martinez et al.~\cite{martinez2017simple} construct a simple fully connected network using only 2D keypoints as input and achieve promising 3D HPE performance, showing that by using 2D joint positions, the 3D human pose can also be efficiently and accurately estimated. Inspired by them and considering the thoroughly investigated 2D HPE, many works decompose the problem into two subtasks, i.e., the 2D HPE and 2D-to-3D pose lifting~\cite{liu2020comprehensive, zeng2021learning, zou2021modulated, xu2021graph}. Therewith, 2D-to-3D pose lifting springs up as a fundamental problem in this area that our work devotes to.

However, human skeleton topology is inherently sparse and graph-structured. Fully connected neural networks are less effective in modeling graph-structured data due to their simple connections among all nodes and the probability of over-fitting. To leverage the information of the human skeletal topology, some works~\cite{ci2019LCN, xu2021graph, zeng2021learning, zou2021modulated} propose to model the human body with GCN and have achieved SOTA results.
For example, Ci et al.~\cite{ci2019LCN} introduced a locally connected network to enhance the representation capability of GCN. Liu et al.~\cite{liu2020comprehensive} explored the weight sharing and feature transformation that occurs before or after feature aggregation in GCN. Nonetheless, existing GCN-based 3D HPE methods update the node features by aggregating their neighbors' information without considering the different contributions of those nodes in different motion patterns.

Instead of considering all the joints of a skeleton as a whole, Xue et al.~\cite{xue2022partaware} demonstrated that the human skeleton exhibits obvious part-wise inconsistency of motion patterns, which is also reported in SRNet~\cite{zeng2020srnet}. However, these works are limited to priory structural information of limb groupings and ignore investigating the latent groups underlying human joint synergies. Besides bio-connection between joints, other complex synergies exist between joints, such as maintaining balance. Moreover, joints in a limb are utilized as a whole to calculate the relationship with other body parts in previous works, which results in a lower accuracy of those peripheral joints, such as wrists and feet.

To address the problems mentioned above in monocular 3D HPE, we propose a novel architecture: Hop-wise GraphFormer with Intragroup Joint Refinement (HopFIR).
The first key component of HopFIR is a novel Hop-wise GraphFormer (HGF) module to employ k-hop neighbors. In HGF, the information of every hop of every joint is aggregated into the hidden space, i.e. we get $N \times k$ groups of features for a skeleton model with $N$ joints. Meanwhile, a hop-wise transformer-like attention mechanism is designed to extract the correlation among feature groups, which computes the similarity by the dot product of the node feature and the group feature. The proposed HGF enables the network to discover latent joint interactions in human joint synergy. Because HGF leverage less prior information about the human body and ignores the interaction among joints in a limb, especially the interaction of peripheral joints associated with a limb. In this regard, we introduced an Intragroup Joint Refinement (IJR) module to strengthen the correlation of intra-group joints grouped by limb prior information. Moreover, a residual block, the HopFIR block is built by two HGF modules, followed by one IJR module. The proposed architecture achieves optimal regression accuracy with a stack of three HopFIR blocks.

In conclusion, our work makes the following contributions:

$\bullet$ To the best of our knowledge, we design the first Hop-wise GraphFormer module to explore potential joint correlations underlying human joint synergy. We also proved that other GCN-based methods can benefit from the proposed HGF module efficiently as shown in Fig.~\ref{fig:overview}.

$\bullet$ We design an Intragroup Joint Refinement module, which attends intragroup joints to refine joint features through the associated limb, especially the wrists and feet. The IJR modules promote HGF modules to discover the latent synergies among joints.

$\bullet$ We proposed a novel Hop-wise GraphFormer with Intragroup Joint Refinement (HopFIR) for 3D HPE, which is built entirely with HGF modules and IJR modules. Specifically, two HGF modules and one IJR module are coupled into a HopFIR block.

$\bullet$ Extensive experiments demonstrate the effectiveness and generalization ability of the proposed modules and HopFIR architecture by providing new state-of-the-art results on two challenging datasets, i.e., Human3.6M~\cite{ionescu2013h36m} and MPI-INF-3DHP~\cite{mehta20173dhp}.

\section{Related Work}
\label{sec:related work}

\textbf{3D Human Pose Estimation.}
Early works~\cite{ramakrishna2012reconstructing, sminchisescu20083d} used handcrafted features, perspective relationships, and geometric constraints to estimate the 3D human pose.
Recent pose estimation approaches can be generally divided into two categories.
The first category of networks regresses 3D human joints from the image directly~\cite{pavlakos2017coarse, zhou2017towards}. Pavlakos et al.~\cite{pavlakos2017coarse} exploited CNN to predict voxel-wise likelihoods for each joint. Zhou et al.~\cite{zhou2017towards} directly embedded a kinematic object model into the networks to learn the general multi-articulate object pose.
Approaches in the second category
decouple the 3D HPE task into 2D pose estimation from an image and 3D pose estimation from the detected 2D joints (2D-to-3D). For example, Martinez et al.~\cite{martinez2017simple} proposed a simple yet effective baseline with fully-connect networks and prove that 3D human poses can be regressed simply and effectively from 2D keypoints. Our paper follows this pipeline and focuses on the 2D-to-3D pose lifting.
Instead of treating all joints of a skeleton as a whole, grouping the joints with consideration of their interactions is crucial in promoting the 3D human pose regression accuracy.
Xue et al.~\cite{xue2022partaware} divided the human skeleton graph into five groups according to the limbs to explore part-wise motion inconsistency.
Zeng et al.~\cite{zeng2020srnet} split the human joints into local regions and recombine the global information from the rest of the joints.
Different from the approaches mentioned above, HopFIR groups joints by k-hop of each joint and prior limb information, which enables the network to discover latent connections between groups in different human joint synergy.

\textbf{Graph Neural Network.}
GCNs~\cite{defferrard2016convolutional, gori2005new, kipf2016semi, scarselli2008graph} generalize the capability of CNNs by performing convolution operations on graph-structured data. Existing GCNs can be divided into two categories: the spectral-based approaches~\cite{defferrard2016convolutional} and the spatial-based approaches~\cite{kipf2016semi}.
Our approach falls into the second category, which applies message-passing operations on the graph nodes and their neighbors.

Due to the graph-structure topology of the human skeleton, many works~\cite{ci2019LCN, liu2020comprehensive} introduce GCN to tackle the 3D HPE task.
Zhao et al.~\cite{zhao2019semantic} proposed a SemGCN to learn the semantic relationships between human joints.
Zou et al.~\cite{zou2021modulated} proposed a weight modulation and an affinity modulation based on SemGCN.
These above methods aggregate the first-order neighborhood messages to update the feature matrix by assigning different weights for different nodes.
Some works~\cite{zhu2021posegtac, zou2020highorder} extend the first-order neighbors to high-order neighbors in the spatial domain directly.
Zeng et al.~\cite{zeng2021learning} designed a hierarchical fusion block by dividing the fusion procedure into two stages, where all the high-order neighbors of a node are aggregated into a feature in the first stage and fuse it with the node feature and the first-order neighbor in the second stage.
Zhao et al.~\cite{zhao2022graformer} introduce Chebyshev graph convolution to fuse information among the k-hop neighbors of a joint directly.
All of them update features by aggregating each node's own k-hop neighborhood information in a GCN layer. However, HopFIR considers the k-hop groups of all nodes to reconstruct the k-hop feature of a node through the proposed attention mechanism, which can enhance the representation capability of GCNs.

\textbf{Graph Attention.}
Graph attention networks~\cite{velivckovic2017gat} is a pioneer work that pays attention to the data in graph structure by assigning attention weight for each node. Recently, transformer~\cite{vaswani2017attention} is proposed for machine translation tasks, proving the capacity of attention on sequence input. ViT~\cite{dosovitskiy2020vit} introduces the Transformer into computer vision and achieves excellent performance. Inspired by them, some researchers further introduce the Transformer for 3D HPE.
Zheng et al.~\cite{zheng2021poseformer} designed a spatial-temporal model to model the human joint relations, which treats each 2D pose as an input token in Temporal Transformer Module and considers each joint as a patch in Spatial Module.
Xue et al.~\cite{xue2022partaware} presented the temporal transformer with a time window and the multi-head attention between the limb group and its part-wise dictionary feature.
Zhao et al.~\cite{zhao2022graformer} applied self-attention to capture global information by calculating the similarity of all nodes.

Different from the existing graph attention mechanism for 3D HPE, we propose the intergroup multi-head attention mechanism among the k-hop groups of all nodes, which assign the attention weights by computing the similarity between the node feature and k-hop group feature. Moreover, we introduce the intragroup multi-head self-attention in limb groups to refine the joint features and promote the HGF to discover the latent synergies among joints.

\section{The proposed HopFIR}
\label{sec:formatting}
This paper proposes a novel architecture to regress 3D human pose from $N$ given 2D keypoints $X\in\mathbb{R}^{N\times2}$. The proposed framework mainly consists of Hop-wise GraphFormer (HGF) modules and Intragroup Joint Refinement (IJR) modules. In the following, we first review the vanilla Graph Convolutional Network and Transformer in Sec.~\ref{subsec:vanilla}. Then, the HGF module and IJR module are introduced in Sec.~\ref{subsec:HGF} and Sec.~\ref{subsec:IJR}, respectively. Finally, we present the network architecture in Sec.~\ref{subsec:architecture}.
\begin{figure}[t]
  \centering

   \includegraphics[width=1\linewidth]{group3.pdf}
   \caption{The illustration of the human skeletal graph and the groups in HGF and IJR modules. The k-hop neighbors of a joint are set as a group in HGF modules where (a) indicates the 1-hop groups of different joints and (b) indicates k-hop groups of joint 9. (c) indicates the joints grouped by prior limb information.
   }
   \label{fig:group}
\end{figure}

\subsection{Vanilla GCN and Transformer}
\label{subsec:vanilla}
\textbf{GCN.} A graph is defined as $\mathcal{G} =(\mathcal{V}, \mathcal{E})$ where $\mathcal{V}$ is a set of N nodes and $\mathcal{E}$ is the adjacency matrix representing edges between the nodes. Given a collection of input features $H^l \in \mathbb{R}^{N \times D}$,
a generic GCN layer that aggregates neighborhood information can be formulated as follows:
\begin{equation}
\label{e1}
    H^{(l+1)}=\sigma(\tilde{A} H^l W)
\end{equation}
where $W\in\mathbb{R}^{D \times D'}$is the learnable weight matrix which transforms the feature dimension from $D$ to $D'$,
and $\sigma(.)$ is the activation function, such as ReLU~\cite{nair2010relu}. $H^{(l+1)}$ is the updated feature matrix,
$\tilde{A}\in \mathbb{R}^{N\times N}$ is the symmetrically normalized affinity matrix~\cite{kipf2016semi} with added self-connections,
$A \in \{0,1\}^{N \times N}$ is the adjacency matrix. The $(i, j)th$ entry $a_{ij}=1$ represent node $j$ is the neighbor of node $i$. Otherwise, they are not connected and $a_{ij}=0$. Therefore, the none-neighbor nodes have a weak influence on each other in vanilla GCN, which hinders the modeling of underlying motion connections in 3D HPE.

\textbf{Transformer.} Transformer architecture relies entirely on self-attention to compute representations of its input and output. The self-attention function maps the inputs to the queries $Q$, keys $K$, and values $V$ by weight matrices $W_Q$, $W_K$, and $W_V$, respectively, and the matrix of outputs is calculated as:
\begin{equation}
    Attention(Q, K, V)=Softmax(QK^T/\sqrt{d})V
\end{equation}
where $d$ is the feature dimension of $Q$, and $\frac{1}{\sqrt{d}}$ is a scaling factor to prevent extremely small gradients.
The multi-head attention which performs self-attention in parallel projects the queries, keys, and values $P$ times with different linear projections to each subspace, respectively.
\begin{equation}
    MultiHead(Q, K, V) = Concat(D_1,...,D_P)W_{o}
\end{equation}
where $W_O$ is the projection matrix of outputs, $D_p = Attention(QW_Q^p, KW_K^p, VW_V^p)$, and $p\in [1,..,P].$

\subsection{Hop-wise GraphFormer}
\label{subsec:HGF}
Previous GCNs studies just aggregate multi-hop neighborhood information~\cite{zeng2021learning, zou2020highorder} or assign an attention weight to each one-order neighbor~\cite{zhao2019semantic,zou2021modulated} for 3DHPE. To effectively capture the node's neighborhood message and increase the representational capacity of GCNs, we introduce the hop-wise GraphFormer (HGF) module which treats each hop as a group (Fig.\ref{fig:group} a,b) and computes the attention weights for each hop. By considering the relationship within $k$ hops, we can obtain $N \times k$ groups for a skeleton graph with $N$ joints, which provides enough combinations of joints for discovering the latent correlations among joints in different human joint synergy.

We first define the k-hop matrix $A^k$ as:
\begin{equation}
    a^k_{ij} = \begin{cases}
    1, &d(v_i, v_j)=k\\
    0, &otherwise
    \end{cases}
\end{equation}
where $d(v_i, v_j)$ denotes the distance of the shortest path between $v_i$ and $v_j$ on the skeleton graph.
The k-hop neighborhood information is aggregated with a weighted sum of the target node's k-hop neighbors which is named HopGCN:
\begin{equation}
\label{e5}
    s^k_i = \sum_{j}a^k_{ij}h_jW^k
\end{equation}
where $s^k_i \in \mathbb{R}^{D}$ is a hidden representation of the k-hop neighborhood information.
Eq.~\ref{e5} is similar to Eq.~\ref{e1} but with an extended definition of $A$ to $k$ hops. The weight matrix $W$ is assigned to each hop respectively.

Before aggregating hidden representation $s^k_i$ to the target node, we propose a transformer-like attention mechanism computing the similarity between the node feature $h_i$ and k-hop hidden representation, as shown in Eq.~\ref{e7}:
\begin{equation}
\label{e7}
    z^k_i=\sum_jsoftmax_j(\frac{h_i^Ts^k_j}{\sqrt{d}})s^k_j.
\end{equation}
Due to the structure of the human skeleton and the joint synergy, the k-hop neighborhood of joint $i$ is related to that of other joints. For example, the synergy of hop $S_5^1$ is related to hop $S_6^1$ and $S_3^1$, as shown in Fig.~\ref{fig:group}. By packing together $z^k_i$, $h_i$ and $s^k_i$ into matrices $Z^k$, $H$ and $S^k$, $Z^k$ can be calculated in parallel:
\begin{equation}
    Z^k=softmax(\frac{H{S^k}^T}{\sqrt{d}})S^k
\end{equation}
Note that one can also reconstruct $Z^k$ using the self-attention mechanism purely on node features or hop features. The performance comparison of these cases is discussed in the experimental results.

We further reduce the dimension of hidden representation by a fully connected layer based on the order of the neighborhood, considering different amounts of information related to the target node, as formulated in Eq.~\ref{eq:fc}.
\begin{equation}\label{eq:fc}
    r^k_i=F^k(z^k_i)
\end{equation}
where $r^k_i$ is the refined representation of the k-hop neighborhood information with respect to node $i$, and $F^k$ is the mapping function. Then, the refined representation is concatenated to the updated node feature.

All the refined representations of hop-wise neighborhoods and the target node feature $h_i$ are encoded to a D-dimensional vector:
\begin{equation}
\label{e10}
    h'_{i}=F(h_i, r_i^1, r_i^2, r_i^3, ..., r_i^k)
\end{equation}
where $h'_i$ is the final updated feature of node $i$ in this layer, and $F$ is the aggregation function.
Packing together the updated features of all nodes, Eq.~\ref{e10} can be rewritten as:
\begin{equation}
    H' = F(H, R^1, R^2, R^3, ..., R^k).
\end{equation}
Thus, we get the core layer of the HGF module, the Hop-wise attention(HA) layer, which extracts latent correlations between feature groups and aggregates k-hop neighborhood information.
With the increment of $k$,
we get more $N$ groups to explore the motion patterns underlying those groups.
One can adjust $k$ based on the tasks and pipelines. In HopFIR architecture, three hops are enough to achieve optimal performance according to the experimental results.

To fuse the current global information and original 2D information, we concatenate all the joint features in a batch and feed them into a fully connected layer to extract current global information. Then we concatenate the extracted global information, 2D information, and the output of HA on the feature dimension of the joint feature to fuse all the information by another fully connected layer.
\begin{figure*}[t]
  \centering

  \includegraphics[width=0.9\linewidth]{framework11.pdf}
   \caption{The HopFIR architecture and the details of HGF modules and IJR modules. The HopFIR block is a residual block~\cite{he2016resnet} built by two HGF modules, followed by one IJR module. The proposed architecture with three blocks achieves optimal performance.}
   \label{fig:framework}
\end{figure*}

\subsection{Intragroup Joint Refinement}
\label{subsec:IJR}
The proposed HGFs split the skeleton graph into different groups based on the k-hop neighborhood of each joint, which pays attention to the key groups in different joint synergies. However,
HGF leverages less prior information about the human body and ignores the interaction among joints in a limb,
especially the interaction of peripheral joints associated with limbs, such as wrists and feet.
In this regard, we introduce an Intragroup Joint Refinement (IJR) module to strengthen the correlation of intra-group joints grouped by limb prior information as shown in Fig.~\ref{fig:group}(c). The HGF features of joints in each limb group are refined by using multi-head self-attention~\cite{vaswani2017attention} in the IJR modules, as denoted in Eq.~\ref{eq:mhsa}.
\begin{equation}
\label{eq:mhsa}
    H_g = MHSA(MHSA(H)_g)
\end{equation}
where $H$ is the feature matrix from HGF module, $MHSA(H)_g$ is the feature matrix of the group $g$ updated by global multi-head self-attention, and $H_g$ is the final feature matrix of the group $g$ updated by IJR module. More details of the IJR module are shown in Fig.~\ref{fig:framework}.
\subsection{HopFIR Architecture}
\label{subsec:architecture}
The HopFIR architecture consists of the proposed Hop-wise GraphFormer (HGF) modules and Intragroup Joint Refinement (IJR) modules, as illustrated in Fig.~\ref{fig:framework}. The residual HopFIR block which contains two HGFs and one IJR module is designed as the basic block in HopFIR.
In the HGF module, the input is first normalized by a layer normalization~\cite{ba2016ln}. Then, the operations described in Sec.~\ref{subsec:HGF} is performed through HopGCN, HA, FC and information fusion in the HGF module, respectively.
After a batch normalization~\cite{ioffe2015bn} and a PReLU~\cite{he2015prelu} activation function, the output is residual connected with the normalized input from the layer normalization.
Moreover, we define an embedding layer to map the input to the latent space and a HG module to transform the output into 3D space.
The HG module is a variant of HGF designed for the output layer, more details are shown in the supplementary material.
The whole HopFIR accepts 2D keypoints as the input, which can be obtained via an off-the-shelf 2D detector.

The graph is obtained by adding a normalized globally learnable k-hop graph to the skeleton graph and then symmetrically normalizing it like~\cite{zou2021modulated}.
We use the L1-norm loss and L2-norm loss to compute the error between ground truth and prediction with the weighed summation as follows:
\begin{equation}
    L = \alpha \sum_{n=1}^N \Vert Y_n - \hat{Y_n} \Vert_2 + \beta \sum_{n=1}^N \Vert Y_n - \hat{Y_n} \Vert_1
\end{equation}
where N is the joint number, $\hat{Y_n}$ is the predicted 3D position of joint $n$, $Y_n$ is the ground truth, $\alpha$ = 1 and $\beta$ = 0.1.

\begin{table*}[t]
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{l c c c c c c c c c c c c c c c c}
        \toprule
        \textbf{Protocol \#1} & Dir. & Disc. & Eat & Greet & Phone & Photo & Pose & Pur. & Sit & SitD. & Smoke & Wait & WalkD. & Walk & WalkT. & Avg.\\
        \midrule\midrule
        Martinez et al.~\cite{martinez2017simple} ICCV'2017& 51.8&  56.2&  58.1&  59.0& 69.5 & 78.4 & 55.2 & 58.1 & 74.0 & 94.6 & 62.3 & 59.1 & 65.1 & 49.5 & 52.4 & 62.9\\

        Zhao et al.~\cite{zhao2019semantic} CVPR'2019& 47.3 & 60.7 & 51.4 & 60.5 & 61.1 &\textbf{49.9} &\underline{47.3} & 68.1 & 86.2 &\textbf{55.0} & 67.8 & 61.0 &\textbf{42.1} & 60.6 & 45.3 & 57.6\\
        Ci et al.~\cite{ci2019LCN} ICCV'2019& 46.8 & 52.3 &\textbf{44.7} & 50.4 & 52.9 & 68.9 & 49.6 & 46.4 & 60.2 & 78.9 & 51.2 & 50.0 & 54.8 & 40.4 & 43.3 & 52.7\\
        Liu et al.~\cite{liu2020comprehensive} ECCV'2020& 46.3 & 52.2 & 47.3 & 50.7 & 55.5 & 67.1 & 49.2 & 46.0 & 60.4 & 71.1 & 51.5 & 50.1 & 54.5 & 40.3 & 43.7 & 52.4\\
        Xu et al.~\cite{xu2021graph} CVPR'2021& 45.2 & 49.9 & 47.5 & 50.9 & 54.9 & 66.1 & 48.5 & 46.3 & 59.7 & 71.5 & 51.4 & 48.6 & 53.9 & 39.9 & 44.1 & 51.9\\
        Zhao et al.~\cite{zhao2022graformer} CVPR'2022& 45.2 & 50.8 & 48.0 & 50.0 & 54.9 & 65.0 & 48.2 & 47.1 & 60.2 & 70.0 & 51.6 & 48.7 & 54.1 & 39.7 & 43.1 & 51.8\\
        Pavllo et al.~\cite{pavllo20193d} CVPR'2019&47.1 &50.6 &49.0 &51.8 &53.6 &61.4 &49.4& 47.4 &59.3& 67.4 &52.4 &49.5& 55.3 &39.5& 42.7 &51.8\\
        Cai et al.~\cite{cai2019exploiting} ICCV'2019&46.5 &48.8 &47.6 &50.9& 52.9& 61.3& 48.3& 45.8& 59.2& 64.4 &51.2 &48.4 &53.5 &39.2 &41.2 &50.6\\
        zeng et al.~\cite{zeng2020srnet} ECCV'2020&\underline{44.5} &\underline{48.2} & 47.1 & \textbf{47.8} & 51.2 &\underline{56.8} & 50.1 &\underline{45.6} & 59.9 & 66.4 & 52.1 &\textbf{45.3} & 54.2 & 39.1 &\underline{40.3} & 49.9\\
        Zou et al.~\cite{zou2021modulated} ICCV'2021& 45.4 & 49.2 & 45.7 & 49.4 &\underline{50.4} & 58.2 & 47.9 & 46.0 &\underline{57.5} & 63.0 &\underline{49.7} & 46.6 & 52.2 &\underline{38.9} & 40.8 &\underline{49.4}\\

        \midrule

        Ours &\textbf{43.9} &\textbf{47.6} &\underline{45.5} &\underline{48.9} &\textbf{50.1} &58.0 &\textbf{46.2} &\textbf{44.5} &\textbf{55.7} &\underline{62.9} &\textbf{49.0} &\underline{45.8} &\underline{51.8} &\textbf{38.0} &\textbf{39.9} &\textbf{48.5}\\
        \bottomrule
        \textbf{Protocol \#2} & Dir. & Disc. & Eat & Greet & Phone & Photo & Pose & Pur. & Sit & SitD. & Smoke & Wait & WalkD. & Walk & WalkT. & Avg.\\
        \midrule\midrule
        Martinez et al.~\cite{martinez2017simple} ICCV'2017& 39.5 & 43.2 & 46.4 & 47.0 & 51.0 & 56.0 & 41.4 & 40.6 & 56.5 & 69.4 & 49.2 & 45.0 & 49.5 & 38.0 & 43.1 & 47.7\\

        Ci et al.~\cite{ci2019LCN} ICCV'2019& 36.9 & 41.6 & 38.0 & 41.0 & 41.9 & 51.1 & 38.2 & 37.6 & 49.1 & 62.1 & 43.1 & 39.9 & 43.5 & 32.2 & 37.0 & 42.2\\
        Liu et al.~\cite{liu2020comprehensive} ECCV'2020& 35.9 & 40.0 & 38.0 & 41.5 & 42.5 & 51.4 & 37.8 & 36.0 & 48.6 & 56.6 & 41.8 & 38.3 & 42.7 & 31.7 & 36.2 & 41.2\\
        Cai et al.~\cite{cai2019exploiting} ICCV'2019& 36.8 & 38.7 & 38.2 & 41.7 & 40.7 & 46.8 & 37.9 & 35.6 & 47.6 &\underline{51.7} & 41.3 & 36.8 & 42.7 & 31.0 & 34.7 & 40.2\\
        Pavllo et al.~\cite{pavllo20193d} CVPR'2019& 36.0 & 38.7 & 38.0 & 41.7 & 40.1 & 45.9 & 37.1 & \underline{35.4} & 46.8 & 53.4 & 41.4 & 36.9 & 43.1 &\textbf{30.3} & 34.8 & 40.0\\
        zeng et al.~\cite{zeng2020srnet} ECCV'2020&\underline{35.8} & 39.2 &\underline{36.6} &\textbf{36.9} & 39.8 & 45.1 & 38.4 & 36.9 & 47.7 & 54.4 &\textbf{38.6} & 36.3 &\textbf{39.4} & \textbf{30.3} & 35.4 & 39.4\\
        Zou et al.~\cite{zou2021modulated} ICCV'2021&\textbf{35.7} &\underline{38.6} &\textbf{36.3} & \underline{40.5} &\textbf{39.2} &\textbf{44.5} &\underline{37.0} &\underline{35.4} &\underline{46.4} &\textbf{51.2} & 40.5 &\underline{35.6} &\underline{41.7} & 30.7 &\underline{33.9} &\underline{39.1}\\
        \midrule

        Ours &\underline{35.8} &\textbf{38.1} &\underline{36.6} & 40.9 &\underline{39.4} &\textbf{44.5} &\textbf{36.7} &\textbf{34.8} &\textbf{45.5} &\underline{51.7} &\underline{40.0} &\textbf{35.5} & 41.8&30.5&\textbf{33.8} &\textbf{39.0}\\
        \bottomrule
    \end{tabular}
    }
    \caption{Quantitative comparison on Human3.6M with detected 2D poses as input under Protocol \#1 and Protocol \#2 in millimeters. The best results are highlighted in bold and the second-best results are presented with underlines.}
    \label{table:2D}
\end{table*}

\begin{table*}[ht]
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{\linewidth}{!}{
    \begin{tabular}{l c c c c c c c c c c c c c c c c}
        \toprule
        \textbf{Protocol \#1} & Dir. & Disc. & Eat & Greet & Phone & Photo & Pose & Pur. & Sit & SitD. & Smoke & Wait & WalkD. & Walk & WalkT. & Avg.\\
        \midrule\midrule
        Zhou et al.~\cite{zhou2019hemlets}(+) ICCV'2019& 34.4 & 42.4 & 36.6 & 42.1 & 38.2 & \textbf{39.8} & 34.7 & 40.2 & 45.6 & 60.8 & 39.0 & 42.6 & 42.0 & 29.8 & 31.7 & 39.9\\
        Ci et al.~\cite{ci2019LCN}(+)($\ast$) ICCV'2019& 36.3 & 38.8 & 29.7 & 37.8 & 34.6 & 42.5 & 39.8 & 32.5 & 36.2 & \textbf{39.5} & 34.4 & 38.4 & 38.2 & 31.3 & 34.2 & 36.3\\
        \midrule
        Martinez et al.~\cite{martinez2017simple} ICCV'2017& 37.7 & 44.4 & 40.3 & 42.1 & 48.2 & 54.9 & 44.4 & 42.1 & 54.6 & 58.0 & 45.1 & 46.4 & 47.6 & 36.4 & 40.4 & 45.5\\
        Zhao et al.~\cite{zhao2019semantic} CVPR'2019& 37.8 & 49.4 & 37.6 & 40.9 & 45.1 & 41.4 & 40.1 & 48.3 & 50.1 & 42.2 & 53.5 & 44.3 & 40.5 & 47.3 & 39.0 & 43.8\\

        Cai et al.~\cite{cai2019exploiting} ICCV'2019& 33.4 & 39.0 & 33.8 & 37.0 & 38.1 & 47.3 & 39.5 & 37.3 & 43.2 & 46.2 & 37.7 & 38.0 & 38.6 & 30.4 & 32.1 & 38.1\\
        Liu et al.~\cite{liu2020comprehensive} ECCV'2020& 36.8 & 40.3 & 33.0 & 36.3 & 37.5 & 45.0 & 39.7 & 34.9 & 40.3 & 47.7 & 37.4 & 38.5 & 38.6 & 29.6 & 32.0 & 37.8\\

        Zou et al.~\cite{zou2021modulated} ICCV'2021& - & - & - & - & - & - & - & - & - & - & - & - & - & - & - & 37.4\\
        zeng et al.~\cite{zeng2020srnet} ECCV'2020& 35.9 & 36.7 & 29.3 & 34.5 & 36.0 & 42.8 & 37.7 & 31.7 & 40.1 & 44.3 & 35.8 & 37.2 & 36.2 & 33.7 & 34.0 & 36.4\\
        Xu et al.~\cite{xu2021graph} CVPR'2021& 35.8 & 38.1 & 31.0 & 35.3 & 35.8 & 43.2 & 37.3 & 31.7 & 38.4 & 45.5 & 35.4 & 36.7 & 36.8 & 27.9 & 30.7 & 35.8\\
        Zhao et al.~\cite{zhao2022graformer} CVPR'2022& 32.0 & 38.0 & 30.4 & 34.4 & 34.7 & 43.3 & 35.2 & 31.4 & 38.0 & 46.2 & 34.2 & 35.7 & 36.1 & 27.4 & 30.6 & 35.2\\
        \midrule
        Ours& \textbf{31.3} & \textbf{34.0} & \textbf{28.0} & \textbf{32.0} & \textbf{33.1} & 42.1 & \textbf{34.1} & \textbf{28.1} & \textbf{33.6} & 39.8 & \textbf{31.7} & \textbf{32.9} & \textbf{33.8} & \textbf{26.7} & \textbf{28.9} & \textbf{32.7}\\
        \bottomrule
    \end{tabular}
    }
    \caption{Quantitative comparison on Human3.6M with under Protocol \#1 in millimeters. Ground truth 2D keypoints are used as input. (+) uses extra data from MPII~\cite{andriluka2014mpii}. ($\ast$) uses pose scales in both training and testing.
    The best results are highlighted in bold.}
    \label{table:GT1}
\end{table*}

\section{Experiments}
In this section, we first introduce the experimental setup and implementation details of the HopFIR networks. Next, we conduct several ablation studies regarding the proposed architecture. Our experimental results and comparisons with state-of-the-art methods are presented in the last.

\subsection{Datasets and Evaluation Protocols}
\textbf{Datesets.}  Human3.6M~\cite{ionescu2013h36m} is currently the largest publicly available dataset for 3D human pose estimation with 3.6 million video frames. It captures the accurate 3D human joint positions from four camera viewpoints and records 11 subjects performing 15 assigned actions. Following previous works~\cite{zou2021modulated, xu2021graph, zhao2019semantic}, we train our model on five subjects (S1, S5, S6, S7, S8) and test on two subjects (S9, S11). In contrast to Human3.6M, MPI-INF-3DHP~\cite{mehta20173dhp} includes complex outdoor scenes, which are usually used to evaluate the generalization ability of the proposed methods. We use the test set of this dataset to verify the generalization of our model.

\textbf{Evaluation Protocols.}  For Human3.6M~\cite{ionescu2013h36m}, we evaluate our model on two standard evaluation protocols, the mean per-joint position error (MPJPE) and the mean per-joint position error after Procrustes alignment (P-MPJPE), which are referred as protocol \#1 and protocol \#2 respectively.
For MPI-INF-3DHP~\cite{mehta20173dhp}, we follow previous works~\cite{zou2021modulated, ci2019LCN, zhao2019semantic} and report the Percentage of Correct Keypoints (PCK) with a threshold of 150mm and the Area Under Curve (AUC) for a range of PCK thresholds.

\subsection{Implementation Details}

We make use of the ground truth 2D joint locations provided in the dataset to align the 3D and 2D poses following the setting of~\cite{zhao2019semantic}.
Moreover, we do not use data augmentation during training with 2D ground truth input, which further verifies the efficacy of our model. Following previous work~\cite{zou2021modulated}, we obtain 2D pose detections using the cascaded pyramid network(CPN)~\cite{chen2018cpn}.
The models are implemented using the PyTorch framework and optimized via Adam~\cite{kingma2014adam}. We initialize the weights in HopFIR using~\cite{glorot2010xavierinit}. All experiments are conducted on a single NVIDIA RTX 3090 GPU.
3D pose regression from 2D detections is more challenging than that from 2D ground truth as the former needs to deal with some extra uncertainty in the 2D space.
Therefore, we set different configurations for them to tackle the uncertainty. In experiments with 2D ground truth as the input, we train the HopFIR networks with an initial learning rate of 0.001, a decay factor of 0.90 per 4 epochs, a batch size of 64, channels of 128, and PReLU activation~\cite{he2015prelu}. When using detected 2D poses, we train the HopFIR networks with the initial learning rate of 0.006, a decay factor of 0.95 per 4 epochs but 0.2 for the first 4 epochs, the batch size of 256, the channel size of 256,  and LeaklyReLU activation~\cite{maas2013leakyrelu}.
Meanwhile, we apply Dropout\cite{JMLR:v15:dropout} with a dropout rate of 0.5 to avoid overfitting.

\subsection{Ablation Study}
\begin{table}[t]
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{75mm}{!}{

    \begin{tabular}{l|c|c|c|c}
        \toprule
        Method & Channels & Params & MPJPE & P-MPJPE \\
        \midrule\midrule
        GCN &128 &0.36M &40.63 &31.65\\
        HopGCN &128 &0.59M &39.15 &31.40\\
        HopGCN + IJR &128 &2.15M &36.62 &29.23\\
        HopGCN + HGF & 128 & 1.05M &35.19 &28.81 \\
        HopGCN + HopFIR & 128 & 2.15M &\textbf{32.67} &\textbf{26.20} \\
        \bottomrule
    \end{tabular}
    }
    \caption{Ablation experiments on the proposed different module.

    }
    \label{table:layer}
\end{table}
\begin{table}[t]
    \small
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{75mm}{!}{

    \begin{tabular}{l|ccc||ccc}
        \toprule
        \multirow{2}{*}{Attention} &\multicolumn{3}{c||}{HA} &\multicolumn{3}{c}{HA+W}\\
        \cline{2-2}\cline{3-3}\cline{4-4}\cline{5-5}\cline{6-6}\cline{7-7}
        &HSS &SSS &SHH &HSS &SSS &SHH\\
        \hhline{-|---||---}
        Params  &\multicolumn{3}{c||}{2.15M} &\multicolumn{3}{c}{2.50M}\\
        \hhline{-|---||---}
        MPJPE   &\textbf{32.67} &34.18 & 34.28 & \textbf{33.29} & 33.53 & 33.90\\
        P-MPJPE &\textbf{26.20} &27.70 & 27.71 & \textbf{27.16} & 27.40 & 27.41\\
        \bottomrule
    \end{tabular}
    }
    \caption{Quantitative comparison of HA layers with different similarity computing approaches.

    }
    \label{table:HA_attn}
\end{table}
\begin{table}[t]
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{83mm}{!}{

    \begin{tabular}{l|c|c|c|c|c}
        \toprule
        \multicolumn{2}{l|}{Method} & Channels & Params & MPJPE & P-MPJPE \\
        \midrule\midrule
        \multicolumn{2}{l|}{SemGCN~\cite{zhao2019semantic}} & 128 & 0.27M & 42.14 & 33.53 \\
        \multicolumn{2}{l|}{SemGCN + HA(HSS)} & 128 & 0.49M &\textbf{38.41} & \textbf{30.56} \\
        \multicolumn{2}{l|}{SemGCN + HA(SSS)} & 128 & 0.49M & 41.30 & 33.07 \\
        \multicolumn{2}{l|}{SemGCN + HA(SHH)} & 128 & 0.49M & 38.81 & 31.05 \\
        \midrule
        \multicolumn{2}{l|}{SemGCN~\cite{zhao2019semantic} w/ Non-local~\cite{wang2018nonlocal}} & 128 & 0.43M & 40.78 & 31.46 \\
        \multicolumn{2}{l|}{SemGCN w/ Non-local +HA(HSS)} & 128 & 0.66M & 38.03 & 30.50 \\
        \multicolumn{2}{l|}{SemGCN w/ Non-local +HA(SSS)} &128 &0.66M &\textbf{37.75} &30.17 \\
        \multicolumn{2}{l|}{SemGCN w/ Non-local +HA(SHH)} &128 &0.66M &37.94 &\textbf{29.71} \\
        \midrule
        \multicolumn{2}{l|}{Modulated GCN~\cite{zou2021modulated}} & 128 & 0.29M & 38.25 & 30.06 \\
        \multicolumn{2}{l|}{Modulated GCN +HA(HSS)+W} & 128 &0.96M &36.54 & 29.09\\
        \multicolumn{2}{l|}{Modulated GCN +HA(SSS)+W} & 128 &0.96M &\textbf{36.14} &\textbf{29.02}\\
        \multicolumn{2}{l|}{Modulated GCN +HA(SHH)+W} & 128 &0.96M &37.38 &30.02\\
        \bottomrule
    \end{tabular}
    }
    \caption{Comparison of the improved performance of proposed HA layer added on different approaches. We test on SemGCN~\cite{zhao2019semantic} and Modulated GCN~\cite{zou2021modulated} that GCN-based method.

    }
    \label{table:HAmodule}
\end{table}

\begin{figure}[t]
  \centering

   \includegraphics[width=0.9\linewidth]{joint1.pdf}
   \caption{Comparison of
   MPJPE of peripheral joints on the test set of the Human3.6M. R and L denote right and left, respectively.}
   \label{fig:joint_error}
\end{figure}

\begin{table}[t]
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{60mm}{!}{
    \begin{tabular}{c|c|c|c|c}
        \toprule
        Num-K & Channels & Params & MPJPE & P-MPJPE \\
        \midrule\midrule
        1 & 128 & 1.88M & 35.88 & 28.76 \\

        2 & 128 & 2.03M & 34.96 & 27.74 \\

        3 & 128 & 2.15M &\textbf{32.67} &\textbf{26.20} \\

        4 & 128 & 2.27M & 35.58 & 28.35  \\
        \bottomrule
    \end{tabular}
    }
    \caption{Ablation study on different number of the k-hop. The units of MPJPE and P-MPJPE are millimeters (mm).}
    \label{table:hopnum}
\end{table}

\begin{table}[t]
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{60mm}{!}{
    \begin{tabular}{c|c|c|c|c}
        \toprule
        Num-Block & Channels & Params & MPJPE & P-MPJPE \\
        \midrule\midrule
        1 & 128 &0.80M &37.33 &30.71\\

        2 & 128 &1.47M &34.21 &27.63 \\

        3 & 128 &2.15M &\textbf{32.67} &\textbf{26.20}\\

        4 & 128 &2.82M &33.84 &27.63\\
        \bottomrule
    \end{tabular}
    }
    \caption{Ablation study on the number of the HopFIR block. The units of MPJPE and P-MPJPE are millimeters (mm).}
    \label{table:block_num}
\end{table}
\begin{table}[t]
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{60mm}{!}{
    \begin{tabular}{c|c|c|c|c}
        \toprule
        Block & Channels & Params & MPJPE & P-MPJPE \\
        \midrule\midrule
        (H)(I) & 128 &1.78M &34.43 &27.79 \\

        (I)(H) & 128 &1.78M &34.13 &27.56 \\

        (H)(I)(H) & 128 & 2.15M &35.08 &27.98 \\

        (I)(H)(H) & 128 &2.15M &34.20 &27.60  \\

        (H)(H)(I) & 128 &2.15M &\textbf{32.67} &\textbf{26.20}\\
        \bottomrule
    \end{tabular}
    }
    \caption{Ablation study on the design of the HopFIR block. (I) is IJR module and (H) is HGF module. The units of MPJPE and P-MPJPE are millimeters (mm).}
    \label{table:block}
\end{table}

The proposed HopFIR contains two main components: Hop-wise GraphFormer (HGF) module and Intragroup Joint Refinement (IJR) module. We conduct a comprehensive ablation study on Human3.6M to validate the effectiveness of each component under controlled settings.
Note that all the input is the 2D ground truth to eliminate the extra uncertainty from the 2D pose detector.

\textbf{Effectiveness of Different Modules.}  We remove IJR and HGF separately to verify the effectiveness of the HGF module and IJR module and conduct experiments on first-order neighbors which remove all modules and the multi-hop mechanism.
In Table~\ref{table:layer},
one can see that each module improves the performance compared to the GCN-only approach. Coupling these two layers to form a HopFIR block achieves further performance improvement.
The HopFIR networks reduce the MPJPE to 32.67mm, which achieves 7.2\% improvement compared to the GraFormer~\cite{zhao2022graformer}.

\textbf{Error on Peripheral Joints.}  We report the regression accuracy of wrists and feet in Fig.~\ref{fig:joint_error}. Compared to some previous methods~\cite{zou2021modulated, zhao2019semantic,zeng2020srnet}, the HopFIR network with IJR modules outperforms previous SOTA methods on all wrists and feet, such as Rfoot 6.7mm, Lfoot 7.7mm, LWrist 9.6mm and RWrist 7.7mm.
The experimental results verify that the intragroup joint attention within in limb group strengthens the capability of the HopFIR.

\textbf{Effectiveness of HA Layer.}  Hop-wise attention is designed to capture interactions among the k-hop groups in different human motion patterns. We calculate the similarity between the group and the target joint to reconstruct the k-hop group feature. Self-attention among groups is an obvious choice. One can also compute the similarity between the node features.
In Table~\ref{table:HA_attn}, we show the experimental results of three different ways to design the HA layer. HSS is the method selected in this paper, where H, S, and S are $Q$, $K$, and $V$, and H and S represent the node feature and the k-hop group feature, respectively. Note that we do not apply a linear transformation of $Q$, $K$, and $V$ like attention~\cite{vaswani2017attention}, so we also show the result of the linear transformation, which is denoted as HA + W. The experiment results show that HSS similarity achieves better performance in the HopFIR architecture, but one can choose the type of similarity according to the network property.

We further insert the HA layer into previous SOTA pose estimation methods to investigate its generalization, such as SemGCN~\cite{zhao2019semantic} and MGCN~\cite{zou2021modulated}. Note that no changes have been made to their source code but insert the HA layer before their information aggregate stage.
The experimental results in Table~\ref{table:HAmodule} show that the HA layer can improve these previous SOTA networks by a large margin, especially SemGCN~\cite{zhao2019semantic}, from 42.14mm to 38.41mm, resulting in a relative 8.9\% improvement.
Experiments on the above methods of aggregating first-order neighbor information demonstrate the effectiveness of HA, which also indicates that latent joint grouping can recognize the latent human joint synergies.

\textbf{Different Number of Hop.}  HopFIR is designed to extract the correlation between feature groups, thus we set different hop $k$ to discover various latent connections underlying human joint synergies, the results are shown in Table~\ref{table:hopnum}.
One can observe that the MPJPE gradually decreases from 35.88 $mm$ to 32.67 $mm$ as the number of hops increases, and reaches the best performance at 3 hops. Therefore, the optimal number of hops is three for 3D HPE, which indicates that we get $16\times3$ groups from the skeleton graph. Each of the 16 groups corresponds to a potential correlation among coupled nodes at different distances, while three is enough to recognize the potential joint synergies.

\textbf{Structure design of the HopFIR block.}  To investigate the structure of the HopFIR block, experiments of HopFIR block number and various combinations of HGF modules and IJR modules were performed.
As the number of blocks increases, the error gradually decreases and reaches the best performance at three.
As shown in Table~\ref{table:block}, (H)(H)(I) achieve the optimal results, which reduces the error to 32.67mm.
In addition, all results except 1 block are better than GraFormer~\cite{zhao2022graformer}, which indicates that the HopFIR has significant human pose representation capability.

\subsection{Comparison with State-of-the-art}
We compare the HopFIR with some state-of-the-art methods on Human3.6M under Protocol\#1 and Protocol\#2 which is shown in Table~\ref{table:2D}.
Our method reaches an MPJPE of 48.50mm. Compared with the previous best results~\cite{zou2021modulated}, we outperform it on all 16 joints.
Due to the uncertainty of 2D detections, we also investigate the capability of HopFIR networks using ground truth 2D key points as input to bypass the influence of detections. As shown in Table~\ref{table:GT1}, HopFIR obtains surprisingly better performance when given precise 2D joint information and attains state-of-the-art performance, which verifies its effectiveness.

\textbf{Cross-Dataset Results on MPI-INF-3DHP.}  Table~\ref{table:mpi3d} further compares HopFIR and previous methods on cross-dataset scenarios to validate its generalization ability. We only train our model on the Human3.6M dataset and test it on the test set of the MPI-INF-3DHP dataset.
The results show that our approach obtains better results compared with other methods, which verifies the generalization ability of our approach to unseen scenarios.

\textbf{Qualitative Results.}  In Fig.~\ref{fig:qualitative}, we show the visual results of HopFIR on Human3.6M in the world space. As we can see, HopFIR networks are able to predict 3D joint positions more accurately, even for difficult poses that MGCN~\cite{zou2021modulated} failed.
\begin{table}[t]
    \centering
    \setlength{\extrarowheight}{0pt}

    \setlength{\aboverulesep}{0pt}
    \setlength{\belowrulesep}{0pt}
    \resizebox{80mm}{!}{
    \begin{tabular}{l|c|c|c|c|c}

        \toprule
        \multirow{2}{*}{Methods}&\multicolumn{4}{c|}{PCK} & \multirow{2}{*}{AUC} \\
        \cline{2-2}\cline{3-3}\cline{4-4}\cline{5-5}
        & GS & no GS & Outdoor & All\\
        \midrule\midrule
        Martinez et al.~\cite{martinez2017simple} & 49.8 & 42.5 & 31.2 & 42.5 & 17.0\\
        Ci et al.~\cite{ci2019LCN} & 74.8 & 70.8 & 77.3 & 74.0 & 36.7\\
        zeng et al.~\cite{zeng2020srnet} & - & - & 80.3 & 77.6 & 43.8\\
        Li et al.~\cite{li2019generating} & 70.1 & 68.2 & 66.6 & 66.9 & -\\
        Zhao et al.~\cite{zhao2022graformer}&80.1 &77.9 &74.1 &79.0 & 43.8\\
        Liu et al.~\cite{liu2020comprehensive} (weight unsharing)& 77.6 & 80.5 & 80.1 & 79.3 & 47.6\\
        Xu et al.~\cite{xu2021graph} & 81.5 & 81.7 & 75.2 & 80.1 & 45.8\\
        Nie et al.~\cite{nie2021lifting} & - & - & - & 83.5 & 45.9\\
        Zou et al.~\cite{zou2021modulated} & 86.4 & \textbf{86.0} &85.7 & 86.1 & 53.7\\
        \midrule

        Ours & \textbf{89.1} & 85.9 & \textbf{85.9} & \textbf{87.2} & \textbf{57.0}\\
        \bottomrule
    \end{tabular}
    }
    \caption{Quantitative comparisons on MPI-INF-3DHP test set. GS denotes green screen.

    }
    \label{table:mpi3d}
\end{table}

\begin{figure}[t]
  \centering
  \includegraphics[width=1\linewidth]{Qualitative4.pdf}

  \caption{Qualitative visual results of our method and MGCN~\cite{zou2021modulated} on Human3.6M dataset~\cite{ionescu2013h36m}. The black lines are Ground Truth (GT) and the red lines are predictions by HopFIR and MGCN.
  }
  \label{fig:qualitative}
\end{figure}
\section{Conclusions}
We present a novel architecture for the 3D human pose estimation, that is, the Hop-wise GraphFormer with Intragroup Joint Refinement. The proposed HGF improves the GCN-based pose estimation networks by grouping the joints by k-hop neighborhood and capturing the potential joint correlations in different joint synergies. Because the peripheral joint strongly interacts with other intra-limb joints, the proposed IJR applies intragroup attention to refining the peripheral joint features through the associated limb. The HopFIR, mainly composed of the HGF modules and the IJR module, achieves new state-of-the-art results while keeping a modest model size.
{\small