text:
\section{Potential negative societal impacts}

\section{Ethics statement}
\label{sec: ethics statement}

Though the purpose of our work is positive as we aim to improve the performance of improve federated learning on non-i.i.d. data, we admit that some adversarial clients may utilize the characteristics of our method to enforce the server to focus on their outlier local gradients, since our method tends to give larger server learning rates when local gradients have more divergent directions. However, safety problems are not in the scope of this paper and belong to another field. One possible solution is applying existing detecting techniques~\citep{krum} to filter out extremely abnormal gradients first before aggregation.

\section{The impact of client sampling when clients perform multiple local updates}
\label{appendix: multiple local steps}

In our main paper, we take the example of FedSGD~\citep{fedavg} to illustrate our claim that, client sampling is the main root cause of the performance degradation of federated learning on non-i.i.d.\ data. In real cases, clients usually perform multiple local updates before uploading the accumulated gradients~\citep{fedavg}, in order to accelerate the convergence. In this case, previous studies~\citep{scaffold} point out that even under the full client participation setting, there still exists the optimization difficulty of FedAvg. The reason is, when the number of local steps increases, the local gradients are accumulated to have more divergent directions, which leads to the unstable aggregation of local gradients in the server. However, this is not contradictory to our claim that, in this scenario, the directions of the averaged server gradients should still be more reliable and accurate under the full client participation setting than that under the partial participation setting.

\section{Discussion about the impact of client sampling on the scale component}
\label{appendix: validate the equal norms of local gradients}

In Eq.~(\ref{eq: optimmal LR approx}) we made the further derivation by claiming that compared with the $GSI$, the scale component $\sqrt{\frac{1}{r} \sum_{k \in S_{t}}\| \boldsymbol{g}^{t}_{k} \|^{2}}$ is barely affected by the client sampling results. We admit that this claim can hold under some certain assumptions. (1) First, if the local optimizer is chosen as Adam, the norm of each local gradient is almost the same. That is because the update scale by using Adam is independent of the scales of gradients, but only depends on the learning rate and the number of updates, which are assumed to be the same across clients. (2) Second, if SGD(M) is the local optimizer, it is true that the scales of local gradients are different across clients, especially under the non-i.i.d. data scenario. However, if the number of local epochs is small, the difference between the norms of local gradients can be negligible compared with the divergence between their directions. This assumption can hold in ours setting since previous studies~\citep{fedprox} suggest to use relatively small number of local epochs under the non-i.i.d. data situation with large number of clients and the small sampling ratio. The reason is, in this case, large local epochs may cause the local model to overfit the local data severely and make the aggregation difficult.

In order to explore the impact of client sampling on the scale component, we conduct the following experiments: we calculate the mean and the standard deviation of the squared norms of the uploaded local gradients in each round,\footnote{The optimization method is FedAvg and the settings are kept as the same as that in the main experiments.} and then calculate the ratio between the standard deviation and the mean value. We report the averaged value of the ratios in each run. If the ratio is small during the training, we can have that the scale component is indeed barely affected by the client sampling.

The results are in Table~\ref{tab: impact on scale component}. As we can see, (1) when the local optimizer is Adam (i.e., experiments on 20NewsGroups), the ratios are all less than 0.2 in all settings, and the ratios are smaller when $\alpha$ is smaller or the number of epochs decreases. This indicates that when Adam is the local optimizer, our assumption that the squared norms of local gradients are almost the same across clients can hold well. (2) When the local optimizer is SGDM (i.e., experiments on CIFAR-10), the ratios become larger as expected. However, even when the number of local epochs is 5 and $\alpha=0.1$, the ratio is smaller than $1/3$, so our assumption that the scale component is barely affected by the client sampling compared with the direction component still roughly holds, and the experimental results also validate the effectiveness of our method even when we choose SGDM as the local optimizer. An interesting phenomenon in CIFAR-10 is, when $\alpha=1.0$, the ratio becomes smaller as the number of local epochs increases, but is still smaller than that when $\alpha=0.1$. Our explanation is, when the number of local epochs is small, the randomness (e.g., data random shuffle, dropout) during the local training plays a great role; and when the number of local epochs is large enough, the scale of each local gradient is stable and only depends on the local training objective.

\section{Visualizations of fluctuation patterns of \texorpdfstring{$GSI$}. on CIFAR-10}
\label{appendix: GSI in CIFAR-10}
In this section, we display the visualizations of fluctuation patterns of $GSI$ in different parameter groups (i.e., weight matrices) during federated training on CIFAR-10~\citep{cifar} with ResNet-56~\citep{resnet}. Results are in Figure~\ref{fig: GSI in CIFAR-10}. The conclusion is the same as that in 20NewsGroups (refer to Figure~\ref{fig: var_term_in_different_param_group}): $GSI$s have different fluctuation degrees in different kinds of parameter groups.

\begin{figure}[t]
\begin{center}
\subfigure{
\begin{minipage}[t]{0.44\linewidth}
\vskip -1.4in
\centering
\makeatletter\def\@captype{table}\makeatother
\caption{The averaged ratio values between the mean and standard deviation of uploaded local gradients in different settings.}
\vskip 0.15in
\scalebox{0.87}{
\begin{tabular}{lcccc}
\toprule
\multirow{2.5}{*}{Dataset} & \multirow{2.5}{*}{$\alpha$} & \multicolumn{3}{c}{Local Epochs}  \\
\cmidrule(lr){3-5}
 & & 1 & 3 & 5\\
\midrule
\multirow{2.5}{*}{CIFAR-10} & 0.1 & 0.24 & 0.29  & 0.33 \\
\cmidrule{2-5}
& 1.0 & 0.23 & 0.19 & 0.15 \\
\midrule
\multirow{2.5}{*}{20NewsGroups} & 0.1 & 0.12  &  0.15 & 0.16\\
\cmidrule{2-5}
& 1.0 & 0.08 & 0.15 & 0.15 \\
\bottomrule
\end{tabular}
}
\label{tab: impact on scale component}
\end{minipage}
}
\subfigure{
\begin{minipage}[t]{0.5\linewidth}
\centerline{\includegraphics[width=1\linewidth]{imgs/cifar10_var_term_E=5.pdf}}
\caption{Fluctuation patterns of $GSI$s in different parameter groups when federated training on CIFAR-10~\citep{cifar} with ResNet-56~\citep{resnet}.}
\label{fig: GSI in CIFAR-10}
\end{minipage}
}
\end{center}
\vskip -0.25in
\end{figure}

\section{Why choose parameter group--wise adaptation}
In Section~\ref{subsec: FedGLAD}, we point out that giving different adaptive learning rates to different parameter groups (i.e., weight matrices) can achieve more precise adjustments compared with the universal adaptation strategy, which considers all parameters as a whole group. We verify the benefit of using the parameter group--wise adaptation strategy in Section~\ref{subsec: the effect of param group-wise}. Researchers can also choose other adaptation strategies, such as adjusting the learning rates layer-wisely that consider all parameters in the same layer as a group. However, we find the layer-wise adaptation performs worse than the parameter group--wise adaptation. The reason is, even in the same layer, $GSI$ has different fluctuation patterns in different kinds of weight matrices (refer to Figure~\ref{fig: var_term_in_different_param_group} and Figure~\ref{fig: GSI in CIFAR-10}). Thus, the parameter group--wise adaptation strategy should be better. On the other hand, we point out that an individual group to be adjusted should contain a certain number of parameters, in order to make the $GSI$ more reliable due to the fact that the neurons in the same global model that are dropped out during different clients' local training are different. This indicates that the element-wise adaptation (i.e., each element in a weight matrix is re-scaled independently based on its own $GSI$) may not work well.

\section{Combining FedGLAD with various federated optimization methods}
\label{appendix: variations}
In our main paper, we put a simplified version of our method when combining it with FedAvg in Algorithm 1. Here, we will give the detailed versions of our method when combined with various optimization frameworks. In the following, we will first illustrate how to apply our method in different kinds of methods.

If the optimization method aims to improve the local training while following the same aggregation mechanism as that of FedAvg, such as FedProx~\citep{fedprox}, SCAFFOLD~\citep{scaffold} or FedDyn~\citep{feddyn}, we can adapt the server learning rates following the same way as that in Algorithm 1.

There are some other methods~\citep{slowmo, fedavgm} which try to utilize a momentum term of the server gradient to help aggregating. In this case, we reformulate the original optimization target in Eq.~(\ref{eq: optimal LR target}) as:
\begin{equation}
\label{eq: optimal LR target with server momentum}
\begin{aligned}
    \eta^{t}_{o} &= \mathop{\arg\min}\limits_{\eta^{t}} \| ( \eta^{t}\boldsymbol{g}^{t} + \beta_{m} \boldsymbol{m}^{t})- (\boldsymbol{g}^{t}_{c} + \beta_{m} \boldsymbol{m}^{t} ) \|^{2}
     = \mathop{\arg\min}\limits_{\eta^{t}} \| \eta^{t}\boldsymbol{g}^{t} - \boldsymbol{g}^{t}_{c}  \|^{2},
        \\ \boldsymbol{g}^{t} &=\frac{1}{r} \sum\limits_{k \in S^{t}} \boldsymbol{g}_{k}^{t}, \quad \boldsymbol{m}^{t}=\eta_{o}^{t-1} \boldsymbol{g}^{t-1} + \boldsymbol{m}^{t-1}.
\end{aligned}
\end{equation}
Thus, the solution can be written as \begin{equation}
\label{eq: optimal LR solution with server momentum}
    \eta^{t}_{o} = \frac{<\boldsymbol{g}^{t}, \boldsymbol{g}_{c}^{t}>}{\| \boldsymbol{g}^{t}\|^{2}},
\end{equation}
which is the same as that in Eq.~(\ref{eq: optimal LR solution}) with $\eta_{s}=1.0$. Therefore, in this case, we can first adjust the current aggregated gradient's scale following the same mechanism discussed through Section~\ref{subsec: FedGLAD}. Then we use the adjusted server gradient to update the momentum, and update the global model with the global learning rate.

\begin{algorithm}[t]
  \caption{Variations of Gradient Similarity--Aware Learning Rate Adaptation for Federated Learning}
  \label{alg: variations of fedglad}
\begin{algorithmic}[1]
  \STATE {\bfseries Server Input:} initial global model $\boldsymbol{\theta}^{0}$, global learning rate $\eta_{0}$, hyper-parameters $\beta=0.9$, $\gamma=0.02$ and $\beta_{1}, \beta_{2}$
  \FOR{each round $t=0$ {\bfseries to} $T$}
  \STATE Sample $r$ clients from $N$ clients: $S^{t} \subset \{1,\cdots,N\}$
  \FOR{$k \in S^{t}$ {\bfseries in parallel}}
  \STATE Broadcast current global model $\boldsymbol{\theta}^{t}$ to client $k$
  \STATE $\boldsymbol{g}_{k}^{t} \leftarrow$ {\bfseries LocalTraining}$(k,\boldsymbol{\theta}^{t}, D_{k}) $
  \ENDFOR
  \STATE $\boldsymbol{g}^{t} \leftarrow \frac{1}{r} \sum\nolimits_{k\in S^{t}} \boldsymbol{g}^{t}_{k}$
  \STATE {\color{red}$\boldsymbol{v}^{t+1} \leftarrow \beta_{2} \boldsymbol{v}^{t} + (1-\beta_{2})\boldsymbol{g}^{t} \odot \boldsymbol{g}^{t} $ (FedAdam)}
  \boxitv{client}{C}{D}
  \FOR{\tikzmk{C} each parameter group $\boldsymbol{\theta}_{P}^{t}$'s gradient $\boldsymbol{g}^{t}_{p}$}

  \STATE $GSI^{t}_{P} \leftarrow \sqrt{( \sum_{k \in S_{t}} \| \boldsymbol{g}^{t}_{P,k} \|^{2} ) / ( r  \| \boldsymbol{g}^{t}_{P} \|^{2}})$
  \IF{$t=0$}
  \STATE $B^{t}_{P} \leftarrow GSI^{t}_{P}$

  \ENDIF
  \STATE $\tilde{\eta}^{t}_{P} \leftarrow \min\{\max\{GSI^{t}_{P} / B^{t}_{P}, 1 - \gamma t\}, 1+\gamma t\} $
  \STATE $B^{t+1}_{P} \leftarrow \beta B^{t}_{P} + (1-\beta) GSI^{t}_{P}$
  \STATE $\boldsymbol{g}^{t}_{P} \leftarrow \tilde{\eta}^{t}_{P} \boldsymbol{g}^{t}_{P} $
  \ENDFOR
  \tikzmk{D}
  \STATE {\color{orange}$\boldsymbol{\theta}^{t+1} \leftarrow \boldsymbol{\theta}^{t} -  \eta_{0} \boldsymbol{g}^{t} $ (FedAvg, FedProx, SCAFFOLD)}
  \STATE $\boldsymbol{m}^{t+1} \leftarrow \beta_{1} \boldsymbol{m}^{t} + (1-\beta_{1})\boldsymbol{g}^{t} $
  \STATE {\color{blue}$\boldsymbol{\theta}^{t+1} \leftarrow \textbf{SGDM}(\boldsymbol{\theta}^{t},\boldsymbol{m}^{t+1},\eta_{0})$ (FedAvgM)}
  \STATE {\color{red}$\boldsymbol{\theta}^{t+1} \leftarrow \textbf{Adam}(\boldsymbol{\theta}^{t},\boldsymbol{m}^{t+1},\boldsymbol{v}^{t+1}, \eta_{0})$ (FedAdam)}
  \ENDFOR
  \STATE
   \STATE {\bfseries Client $k$'s Input:} current global model $\theta^{t}$, local dataset $D_{k}$, local training hyper-parameters
  \STATE {\bfseries LocalTraining}$(k,\boldsymbol{\theta}^{t}, D_{k}) $:
  \STATE Client $k$ performs centralized training to $\boldsymbol{\theta}^{t}$ on local data $D_{k}$ with tuned training hyper-parameters, get final model $\boldsymbol{\theta}_{k}^{t+1}$
  \STATE {\bfseries Return} $\boldsymbol{g}^{t}_{k} = \boldsymbol{\theta}^{t} - \boldsymbol{\theta_{k}}^{t+1}$
\end{algorithmic}
\end{algorithm}

If the federated optimization framework~\citep{fedopt} utilizes an adaptive optimizer such as Adam~\citep{adam} or AdaGrad~\citep{adagrad} to update the global model in the server, our solution is to only adjust the scale of the server gradient used in the numerator while keeping the squared gradient in the denominator unchanged. There are two reasons: (1) The scale of the server gradient will not only affect the first moment estimate $\boldsymbol{m}^{t}$ in the numerator, but also the second moment estimate $\boldsymbol{v}^{t}$ in the denominator. For example, if we decrease the scale of current server gradient, then if the squared gradient $\boldsymbol{g}^{t} \odot \boldsymbol{g}^{t}$ is also correspondingly down-scaled, the second moment estimate $\boldsymbol{v}^{t}$ in the denominator will also decrease, which may lead to larger adaptive learning rate, and this is not we expect to happen. (2) The second reason is, the purpose of using the second moment estimate in Adam is to estimate and adjust the scale of current gradient based on the previous steps' gradients as the training goes on, while the purpose of our method is to optimize the scale of current gradient to mitigate the negative effect brought by the randomness of client sampling at a given round. Thus, we choose to keep the scale of the squared gradient unchanged.

We summarize all above variations in detail in Algorithm~\ref{alg: variations of fedglad}.

\section{Datasets and models}
\label{appendix: datasets and models}
Here we provide the detailed descriptions of the datasets and models we used in our experiments in Table~\ref{tab: datasets and models}. We get data from their public releases. Since these datasets are widely used, they do not contain any personally private information. Some of them do not have the public licenses, so we give all the public links to the datasets we use in the following. Following previous studies~\citep{fedopt}, we use all original training samples for federated training and use test examples for final evaluation. We tune the hyper-parameters based on the averaged training loss of the previous 20\% communication rounds~\citep{fedopt}.

We use the ResNet-56~\citep{resnet} as the global model for CIFAR-10\footnote{CIFAR-10 can be obtained from \href{https://www.cs.toronto.edu/~kriz/cifar.html}{https://www.cs.toronto.edu/~kriz/cifar.html}.}~\citep{cifar}, and the same CNN used in~\citet{fedopt} for MNIST\footnote{MNIST can be obtained from \href{http://yann.lecun.com/exdb/mnist/}{http://yann.lecun.com/exdb/mnist/}.}~\citep{mnist}. For 20NewsGroups\footnote{20NewsGroups can be obtained from \href{http://qwone.com/~jason/20Newsgroups/}{http://qwone.com/~jason/20Newsgroups/}.}~\citep{20news} and AGNews\footnote{The source page of AGNews corpus is \href{http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html}{http://groups.di.unipi.it/~gulli/AG\_corpus\_of\_news\_articles.html}.}~\citep{agnews}, we choose the DistilBERT~\citep{distilbert} as the global model, since previous work~\citep{scaling_FL} validates the great effectiveness of utilizing pre-trained language models for federated fine-tuning, and DistilBERT is a powerful lite version of BERT~\citep{BERT} which makes it more suitable to local devices. The \texttt{max\_seq\_length} is set as 128. We also conduct experiments on BiLSTM~\citep{lstm}, and display the results in Appendix~\ref{appendix: bilstm}.

\input{tables/datasets_and_models}

\section{Data partitioning strategy}
\label{appendix: data partition strategy}

To simulate the non-i.i.d.\ data partitions, we follow FedNLP~\citep{fednlp} by taking the advantage of the Dirichlet distribution. Specifically, assume there are totally $C$ classes in the dataset, the vector $\mathbf{q_{k}}$ ($ q_{k,i} \geq 0,i\in [1, C], \|\mathbf{q_{k}} \|_{1}=1$) represents the label distribution in client $k$, then we draw $\mathbf{q_{k}}$ from a Dirichlet distribution $\mathbf{q_{k}} \sim \text{Dir}(\alpha\mathbf{p})$ where $\mathbf{p}$ represents the label distribution in the whole dataset and $\alpha$ controls the degree of skewness of label distribution: when $\alpha$ becomes smaller, the label distribution becomes more skewed; when $\alpha \rightarrow \infty$, it becomes the i.i.d.\ setting. We assume each client has the same number of local training samples,\footnote{In this paper, we mainly focus on the label distribution shift problem, rather than the quantity shift problem. Also, since we assume each client performs the same number of local updates following previous studies, this practice is reasonable.} so we can allocate samples to client $k$ once $\mathbf{q_{k}}$ is determined. In order to solve the problem that some clients may not have enough examples to sample from specific classes as the sampling procedure goes, FedNLP provides a \textit{dynamic reassigning strategy} that can use samples from other classes which still have remainings to fill the vacancy of the chosen class whose samples are all used out. More details can be found in~\citet{fednlp}.

\section{Baseline methods}
\label{appendix: baseline methods}
We choose 4 widely-used federated optimization methods as baselines in our main experiments, and they are: (1) FedAvg~\citep{fedavg}: the most popular baseline in federated learning; (2) FedProx~\citep{fedprox}: the widely-adopted baseline which aims to optimize the directions of local gradients, while keeping as the same aggregation mechanism as that of FedAvg in the server; (3) FedAvgM~\citep{fedavgm, slowmo}: the method that adds the momentum term during the server's updates; (4) FedAdam~\citep{fedopt}: the method that utilizes the adaptive optimizer Adam~\citep{adam} to update the global model with the averaged server gradient. We also conduct experiments on another baseline SCAFFOLD~\citep{scaffold}, while during experiments we find SCAFFOLD itself can not converge well in our main experiments' settings. Thus, we make a separate discussion about this part in Appendix~\ref{appendix: scaffold}.

\section{Training details}
\label{appendix: training details}
\subsection{Local training}
Firstly, we find SGDM is more suitable to train CNNs and Adam is more suitable to train Transformer-based models. Thus we choose the client optimizer as SGDM when training on two image classification datasets, and choose the client optimizer as Adam for two text classification datasets. Then we perform centralized training on each dataset in order to find the optimal client learning rate $\eta_{l}$ and batch size $B$, and use that learning rate and batch size as the local training hyper-parameters for all federated optimization baselines in the federated training setting. The optimal local training hyper-parameters for each dataset are listed in Table~\ref{tab: local training hyper-parameters}. To decide the number of local training epoches for each dataset, we tune it from $\{1,3,5,10\}$ by using FedAvg, and use the tuned one for other optimization baselines.

\subsection{Server aggregating}
\label{subappendix: server aggregating}
We allow each baseline to tune its own optimal server learning rate in each data partition setting. The server learning rate search grids for all baseline methods are displayed in Table~\ref{tab: baseline lr search grids}. We also display the server learning rates we have tuned and used in our main experiments in Table~\ref{tab: best server lr in our experiments}. The server learning rates used for the baseline methods in Section~\ref{subsec: different sampling ratios} and Section~\ref{subsec: different sampling strategies} are re-tuned, since we find the optimal server learning rates under different sampling ratios or different sampling strategies are different.

There are several reasons to use the above hyper-parameters' tuning strategy: (1) Since clients have local data, the optimal hyper-parameters in the centralized training setting should also work well for clients' local training in the federated training. (2) Using the same local training hyper-parameters for each baseline method allows us to make a fair and clear comparison between them.

\subsection{Infrastructure and training costs}
All experiments are conducted on 4$*$TITAN RTX. The running time of each method on CIFAR-10, MNIST, 20NewsGroups and AGNews is about 12h, 0.5h, 6h and 6h respectively.

\subsection{Hyper-parameter's tuning of FedProx}
\label{appendix: best mu}
For FedProx, we allow it to tune its own hyper-parameter $\mu$ from $\{1.0,0.1,0.01,0.001\}$ in each data partition setting, and we display the best $\mu$ in each setting in Table~\ref{tab: best mu}. As we can see, there is no general $\mu$ for all settings, and researchers may carefully tune this hyper-parameter in each setting.

\subsection{Hyper-parameter's tuning of FedGLAD}
As for our method, the key hyper-parameter is the bounding factor $\gamma$. Though the best $\gamma$ in each setting depends on its dataset, the label skewness degree and the backbone model, in our main experiments we show that there exists a general value (i.e., $\gamma=0.02$) which works well in most settings. If researchers want to tune the optimal $\gamma$ in each setting to achieve better performance, the recommended grid is $\{0.01,0.02,\cdots,0.05\}$.

Since our method aims to adjust the server learning rate relatively, it is also important to choose a proper global learning rate $\eta_{0}$ (refer to Algorithm~\ref{alg: variations of fedglad}).\footnote{This can be an important hyper-parameter if the optimization method follows the same aggregation strategy as that of FedAvg. However, if using FedAvgM or FedAdam, $\eta_{0}=1.0$.} During our experiments, when applying our method, we directly use the server learning rate used in the original baseline without extra tuning, since we assume the optimal learning rate for the original baseline is the right choice for the global learning rate after applying our method, and we want to show our method is easy to apply. However, it is expectable that re-tuning the global learning rate for our method may bring more improvement.

\input{tables/centralized_results}

\section{Results of centralized training}
\label{appendix: centralized results}
For comparison with the results in the federated training setting, we display the results on all datasets in the traditional centralized training setting in Table~\ref{tab: centralized results}. The training settings and hyper-parameters on each dataset are kept as the same as that in the local training of the federated learning.

\section{Experiments on \texorpdfstring{$\alpha=1.0$}.}
\label{appendix: alpha=1.0}
Besides the non-i.i.d. degrees we set in our main paper to conduct the main experiments, we also explore the performance of our method when $\alpha=1.0$. The baselines and the settings are the same as that in the main experiments, and the results are in Table~\ref{tab: main results alpha=1.0}.

As we can see, FedGLAD can also consistently bring improvement in this setting, which validates the effectiveness of our method. However, compared with the results in Table~\ref{tab: main results} in the main paper, we find that on each dataset, the overall trend is \textbf{the improvement brought by our method are greater if the non-i.i.d.\ degree is larger (i.e., smaller $\mathbf{\alpha}$)}. Our explanation is when the label distribution is more skewed, the impact of client sampling becomes larger. For example, when data is more non-i.i.d.\ and the sampled clients have the similar but skewed local data distributions, the averaged server gradient's direction will deviate far more away from the ideal direction that we assume is averaged by all clients' local gradients. Thus, it becomes more important to adjust the server learning rates in these scenarios.

\input{tables/main_results_alpha1}

\begin{figure}[t]
\vskip 0.1in
\begin{center}
\subfigure[Results on CIFAR-10.]{   \begin{minipage}[t]{0.48\linewidth}  \centerline{\includegraphics[width=\columnwidth]{imgs/cifar10_bounding_factor_E=5.pdf}}
\end{minipage}  }
\subfigure[Results on 20NewsGroups.]{   \begin{minipage}[t]{0.48\linewidth}
\centerline{\includegraphics[width=\columnwidth]{imgs/20news_bounding_factor.pdf}}
\end{minipage}
}
\caption{The results of using different $\gamma$ for FedGLAD. $\gamma=0$ represents the original baseline and $\gamma=\infty$ represents removing the bounds from the beginning. $\gamma=0.02$ can be a generally good choice.}
\label{fig: sensitivity on gamma}
\end{center}
\end{figure}

\section{The sensitivity to the bounding factor \texorpdfstring{$\gamma$}.}
\label{appendix: sensitivity of bounding factor}
Here, we want to explore our method's sensitivity to the bounding factor $\gamma$. The results are displayed in Figure~\ref{fig: sensitivity on gamma}. Generally speaking, the best $\gamma$ in each setting depends on many factors, such as the dataset and the non-i.i.d.\ degree of data distribution across clients, but from figures, we can see that there exists a general value (i.e., $\gamma=0.02$) which works well in most cases. Therefore, we choose $0.02$ as the default value in the experiments in the main paper. Moreover, (1) though the performance of removing the bounds is better than the original baseline in some cases, it is consistently worse than that of setting the bounds. (2) The standard deviation in each experiment of using larger $\gamma$ increases. This validates our motivation that without the restrictions, the learning rates at the beginning will vary greatly, which will cause harm to the training.

\input{tables/results_on_scaffold}

\section{Experiments on SCAFFOLD}
\label{appendix: scaffold}
Besides the baselines we choose in the main experiments, we also conduct experiments with another baseline SCAFFOLD~\citep{scaffold}. SCAFFOLD aims to optimize the directions of the averaged server gradients by improving the local training with the help of local control variates. The original version of SCAFFOLD assumes the client optimizer is SGD, while we make some adaptations to make it applicable when the client optimizer is SGDM or Adam. However, we find the SCAFFOLD can not converge well on two image classification tasks if we follow the same training settings in our main experiments. We find the key factor which will affect the performance of SCAFFOLD is the sampling ratio and the number of local training epochs $E$. That is, if the sampling ratio is very small and when $E$ is very large, SCAFFOLD may not converge well. We guess the reason may be that, when the sampling ratio is small, the number between two consecutive rounds of participation of client $k$ is large. Thus, the local control variate of client $k$ in the latter round may no longer be reliable, and using this control variate to adjust the directions of local gradients for many local update steps will cause harm to the local training. While for two text classification tasks, we choose $E$ as 1 and also use a pre-trained backbone that already mitigates the problem caused by the non-i.i.d.\ data distribution and the client sampling at a certain degree, so the SCAFFOLD can work well on these two text classification datasets.

We keep the training hyper-parameters the same as that in our main experiments for two text classification tasks. After re-tuning, we choose $E=1$ for two image classification tasks. Other settings are the same as that in our main experiments. We find SCAFFOLD itself can still not converge well on CIFAR-10, so we only report the results on the other three datasets in Table~\ref{tab: results on scaffold}. The conclusions are the same as those in the main experiments: FedGLAD is also applicable with SCAFFOLD and tends to bring more improvement when non-i.i.d.\ degree is larger.

\section{Experiments on BiLSTM}
\label{appendix: bilstm}

\input{tables/results_on_bilstm}

In our main experiments, we use a pre-trained backbone DistilBERT for two text classification tasks. In this section, we conduct extra experiments with a not pre-trained backbone BiLSTM~\citep{lstm} on 20NewsGroups and AGNews. The model is a 1-layer BiLSTM with the hidden size of 300. We use the pre-trained word embeddings \texttt{glove.6b.300d}\footnote{Downloaded from \href{https://nlp.stanford.edu/projects/glove}{https://nlp.stanford.edu/projects/glove}.}~\citep{glove} for the embedding layer. The \texttt{max\_seq\_length} is 128. The dropout ratio for the LSTM layer is 0.2.

We use FedAvg and FedAvgM as baselines, and the non-i.i.d.\ degree $\alpha=0.1$. Other details can be found in Table~\ref{tab: details of bilstm}. The results are in Table~\ref{tab: results on bilstm}. As we can see, since BiLSTM is not pre-trained, the performance gap between the federated training and the centralized training is larger than that of using DistilBERT as the backbone under the same number of communication rounds.
Secondly, after applying our learning rate adaptation mechanism, the performance consistently becomes better. The experimental results validate the effectiveness of our method on the BiLSTM.

\input{tables/results_of_different_local_epochs}

\section{Experiments with different numbers of local epochs}
\label{appendix: different local epochs}
In our main experiments, we set local epochs $E=5$ for image classification tasks. As we discussed in Section~\ref{subsec: analysis of client sampling} in the main paper and Appendix~\ref{appendix: validate the equal norms of local gradients}, when the local optimizer is SGD(M) and when the number of epochs increases, our assumption about the scale component and the direction component may not generally holds. However, also as we explained, increasing local epochs will easily make the local model overfit to the local dataset severely, and the choices of local epochs in our experiments are also widely-adopted choices in previous studies~\citep{fedopt, fednlp}.

Here, we conduct further experiments to explore the impact of the large number of local epochs on FedGLAD. We perform experiments on CIFAR-10 and MNIST with $\alpha=0.1$ and the local epochs $E=10$ and $E=20$. As we can see from the Table~\ref{tab: different local epochs}, FedGLAD can also bring improvement in almost all settings. Our explanation is, in this case, the negative effect of bad sampling results will be further amplified, and GSI can detect this anomaly and decrease the learning rate in time. Also, during training we find that the accuracy curves (also the loss curves) vary greatly when $E=10$ or $E=20$, indicates that they may not be proper choices in ours settings, and $E=5$ can guarantee the stable training.

\begin{figure}[t]

\begin{minipage}{.48\linewidth}
\centering
\makeatletter\def\@captype{table}\makeatother
\caption{Results on each seed under $\alpha=0.1$.}
\label{tab: each seed result alpha=0.1}
\vskip 0.15in
\scalebox{0.77}{
\begin{tabular}{llccc}
\toprule
\multirow{2.5}{*}{Dataset} & \multirow{2.5}{*}{Method} & \multicolumn{3}{c}{Random Seed}  \\
\cmidrule(lr){3-5}
 & & Seed1 & Seed2 & Seed3  \\
\midrule
\multirow{4.5}{*}{CIFAR-10} & FedAvg &  37.08 &33.90& 34.91 \\
& ~~ \small{+ FedGLAD} & \textbf{39.49} & \textbf{38.02}	& \textbf{37.82} \\
\cmidrule{2-5}
 & FedAvgM & 47.24 & 49.38& 49.76  \\
 & ~~ \small{+ FedGLAD} &  \textbf{49.40} & \textbf{51.91} & \textbf{51.42} \\
\cmidrule{1-5}
\multirow{4.5}{*}{MNIST} & FedAvg &  78.29& 77.77 &78.44 \\
& ~~ \small{+ FedGLAD} & \textbf{79.20} & \textbf{79.75} & \textbf{80.19} \\
\cmidrule{2-5}
 & FedAvgM & 80.68 & 82.65 & 81.79  \\
 & ~~ \small{+ FedGLAD} & \textbf{81.68} & \textbf{82.84} & \textbf{81.81} \\
\cmidrule{1-5}
\multirow{4.5}{*}{20NewsGroups} & FedAvg & 72.72 & 72.96 & 72.84\\
& ~~ \small{+ FedGLAD} & \textbf{73.13} & \textbf{73.22} & \textbf{73.06}  \\
\cmidrule{2-5}
 & FedAvgM & 75.39 & 75.38 & 75.56 \\
 & ~~ \small{+ FedGLAD} & \textbf{76.02} & \textbf{76.12} & \textbf{75.58}  \\
 \cmidrule{1-5}
\multirow{4.5}{*}{AGNews} & FedAvg & 78.95 & 79.78 & 79.72 \\
& ~~ \small{+ FedGLAD} & \textbf{79.83} & \textbf{81.29} & \textbf{81.11} \\
\cmidrule{2-5}
 & FedAvgM &82.25 & 79.67 & 78.91 \\
 & ~~ \small{+ FedGLAD} & \textbf{84.22} & \textbf{84.14} & \textbf{82.67}  \\
\bottomrule
\end{tabular}
}
\vskip -0.1in
\end{minipage}
\hfil
\begin{minipage}{.48\linewidth}
\makeatletter\def\@captype{table}\makeatother \centering
\caption{Results on each seed under $\alpha=1.0$.}
\label{tab: each seed result alpha=1.0}
\vskip 0.15in
\scalebox{0.77}{
\begin{tabular}{llccc}
\toprule
\multirow{2.5}{*}{Dataset} & \multirow{2.5}{*}{Method} & \multicolumn{3}{c}{Random Seed}  \\
\cmidrule(lr){3-5}
 & & Seed1 & Seed2 & Seed3  \\
\midrule
\multirow{4.5}{*}{CIFAR-10} & FedAvg & 73.20 & 70.70 & 72.36 \\
& ~~ \small{+ FedGLAD} & \textbf{74.07}	& \textbf{71.91} & \textbf{72.39} \\
\cmidrule{2-5}
 & FedAvgM &  72.18 & 72.44 & 73.26 \\
 & ~~ \small{+ FedGLAD} & \textbf{72.70} & \textbf{73.18} & \textbf{73.32} \\
\cmidrule{1-5}
\multirow{4.5}{*}{MNIST} & FedAvg & 91.48 & 91.30 & 91.49\\
& ~~ \small{+ FedGLAD} & \textbf{91.70}	 & \textbf{91.55} & \textbf{91.65} \\
\cmidrule{2-5}
 & FedAvgM &  93.55 & 93.52 & 93.04 \\
 & ~~ \small{+ FedGLAD} & \textbf{94.06} & \textbf{94.17} & \textbf{93.67} \\
\cmidrule{1-5}
\multirow{4.5}{*}{20NewsGroups} & FedAvg & 76.57 & 76.35 & 77.06 \\
& ~~ \small{+ FedGLAD} & \textbf{77.53} & \textbf{76.74} & \textbf{77.37} \\
\cmidrule{2-5}
 & FedAvgM & 79.51 & 79.20 & 79.82 \\
 & ~~ \small{+ FedGLAD} & \textbf{79.80} & \textbf{79.68} & \textbf{79.83} \\
 \cmidrule{1-5}
\multirow{4.5}{*}{AGNews} & FedAvg & 91.26 & 91.31 & 91.23 \\
& ~~ \small{+ FedGLAD} & \textbf{91.38} & \textbf{91.41} & \textbf{91.34}\\
\cmidrule{2-5}
 & FedAvgM &90.28 & 89.77 & 90.17  \\
 & ~~ \small{+ FedGLAD} &\textbf{90.82} & \textbf{89.99} & \textbf{90.53} \\
\bottomrule
\end{tabular}
}
\vskip -0.1in
\end{minipage}

\end{figure}

\section{Results on each seed of each experiment}
\label{appendix: each seed result}
As we mentioned in our paper in the experiment analysis section (Section~\ref{subsec: main results}), the standard deviation in each experiment in Table~\ref{tab: main results} and Table~\ref{tab: main results alpha=1.0} may seems a bit large compared with the improvement gap. However, we display the result on each seed of our method and baselines (take FedAvg and FedAvgM as examples) in the Table~\ref{tab: each seed result alpha=0.1} and Table~\ref{tab: each seed result alpha=1.0}, we point out that actually \textbf{on almost each seed of one specific experiment, applying our method can gain the improvement over the corresponding baseline.} The large standard deviation of the baseline results are mainly caused by the fact that the large non-i.i.d. degree makes the baseline training sensitive to the random seeds.

\begin{figure}[t]
\vskip 0.1in
\begin{center}
\centerline{\includegraphics[width=0.78\columnwidth]{imgs/gsi_and_data_sim.pdf}}
\caption{GSI is positively  related to the the merged data distribution
similarity between the sampled clients and all clients.
}
\label{fig: correlation of GSI and data sim}
\end{center}
\vskip -0.1in
\end{figure}

\section{Correlation between \texorpdfstring{$GSI$}. and the merged data distribution similarity between the sampled clients and all clients}
\label{appendix: correlation between GSI and data sim}
Here, we make deep analysis about the correlation between our proposed $GSI$ and the merged data distribution similarity between the sampled clients and all clients.
First, we calculate the $GSI^{t}$ in each round considering all parameters as a group. At the same time, we calculate the similarity score between the merged data distribution of currently sampled clients and that of all clients. The similarity score is based on the cosine similarity between two distributions: denote $L$ as the total number of classes, $n_{k}^{(i)}$ as the number of samples belonging to label $i$ in client $k$'s local data, $n_{k}=\sum\limits_{i=i}^{L} n_{k}^{i}$, then the merged data distribution of sampled clients is $d^{t}=(\frac{\sum\limits_{k\in S^{t}} n^{(1)}_{k}}{\sum\limits_{k\in S^{t}} n_{k}}, \cdots, \frac{\sum\limits_{k\in S^{t}} n^{(L)}_{k}}{\sum\limits_{k\in S^{t}} n_{k}})$. Similarly, the merged data distribution of all clients is written as $d_{c}=(\frac{\sum\limits_{k=1}^{N} n^{(1)}_{k}}{\sum\limits_{k=1}^{N} n_{k}}, \cdots, \frac{\sum\limits_{k=1}^{N} n^{(L)}_{k}}{\sum\limits_{k=1}^{N} n_{k}})$. Then, the similarity score can be calculated as
$$
sim\_score^{t}=\texttt{CosSim}(d^{t}, d_{c}).
$$

Then we visualize the scatter plot of the corresponding $GSI^{t}$ and $sim\_score^{t}$ on CIFAR-10 in Figure~\ref{fig: correlation of GSI and data sim}. The red line corresponds to the linear regression result. The Pearson correlation coefficient between two variables is 0.4753, revealing that \textbf{GSI is positively related to the the merged data distribution similarity between the sampled clients and all clients}. This indicates that the GSI can effectively reflect the sampling results, and then be used to optimize the server learning rates, which helps to verify our original motivation.

\newpage
\input{tables/best_mu}

\input{tables/local_training_hyperparameters}

\input{tables/baseline_lr_search_grids}

\input{tables/best_server_lr_in_our_experiments}

\input{tables/hyperparams_for_bilstm}

\section{Introduction}

\label{sec: intro}
As tremendous data is produced in various edge devices (e.g., mobile phones) every day, it becomes important to study how to effectively utilize the data without revealing personal information and privacy. Federated Learning~\citep{FL, fedavg} is then proposed to allow many clients to jointly train a well-behaved global model without exposing their private data. In each communication round, clients get the global model from a server and train the model locally on their own data for multiple steps. Then they upload the accumulated gradients
only to the server, which is responsible to aggregate (average) the collected gradients and update the global model. By doing so, the training data never leaves the local devices.

It has been shown that the federated learning algorithms perform poorly when training samples are not independently and identically distributed (non-i.i.d.) across clients~\citep{fedavg, federated_empirical_study}, which is the common case in reality. Previous studies~\citep{weight_divergence, scaffold} mainly attribute this problem to the fact that the non-i.i.d.\ data distribution leads to the divergence of the directions of the local gradients. Thus, they aim to solve this issue by making the local gradients have more consistent directions~\citep{fedprox, fedaux, feddyn}. However, we point out that the above studies overlook the negative impact brought by the client sampling procedure~\citep{fedavg, impact_client_sampling}, whose existence we think should be the main cause of the optimization difficulty of the federated learning on non-i.i.d.\ data.

Client sampling is widely applied in the server to solve the communication difficulty between the great number of total clients and the server with limited communication capability, by only sampling a small part of clients to participate in each round. We find that the client sampling induces the negative effect of the non-i.i.d.\ data distribution on federated learning. For example, assume each client performs one full-batch gradient descent and uploads the local full-batch gradient immediately (i.e., FedSGD in~\citet{fedavg}), (1) if all clients participate in the current round, it is equivalent to performing a global full-batch gradient descent on all training samples, and the aggregated server gradient is accurate regardless of whether the data is i.i.d.\ or not; (2) if only a part of clients is sampled for participation, the aggregated server gradient will deviate from the above global full-batch gradient and its direction depends on the merged data distribution of the currently sampled clients (i.e., the label distribution of the dataset constructed by data points from all sampled clients). The analysis is similar in the scenario where clients perform multiple local updates before uploading the gradients, and we have a detailed discussion in Appendix~\ref{appendix: multiple local steps}.

\begin{wrapfigure}{R}{0.52\linewidth}
\begin{center}
\centerline{\includegraphics[width=1.0\linewidth]{imgs/demo.pdf}}
\caption{An illustration of the impact of client sampling through a binary (cats v.s. dogs) image classification task. \textbf{(Right Top)}: If Client 1 and Client 2 are selected, their merged local samples are almost all cat images. Then the averaged gradient deviates far away from the ideal gradient that would be averaged by all clients' local gradients. \textbf{(Right Bottom)}: If Client 1 and Client N are selected, their merged data distribution matches well with the global data distribution merged by all clients' data, and the averaged gradient has a more reliable direction.}
\label{fig: demo}
\end{center}
\end{wrapfigure}

The above analysis indicates that the reliability of the averaged gradients depends on the consistency between the merged data distribution of currently sampled clients and that of all available clients.
Specifically, take the image classification task as an example\footnote{Though in our example in Figure~\ref{fig: demo}, for simplicity, we assume the merged data distribution of all available clients is balanced, we do not have this assumption in our later analysis.} (refer to Figure~\ref{fig: demo}): (1) if local samples from all selected clients are almost all cat images, the averaged gradient's direction deviates far away from the ideal server gradient averaged by all clients' local gradients. In this case, we may not trust the aggregated gradient, and decrease the server learning rate. (2) However, if the merged data distribution of selected clients matches well with the merged data distribution of all clients, the averaged gradient's direction is more reliable. Thus, we should relatively enlarge the server learning rate. This example motivates us to set dynamic server learning rates across rounds instead of a fixed one as previous methods do

In this paper, we first analyze the impact of client sampling and are motivated to mitigate its negative impact by dynamically adjusting the server learning rates based on the reliability of the aggregated server gradients. We theoretically show that the optimal server learning rate in each round is positively related to an indicator called the \textbf{Gradient Similarity--aware Indicator ($\mathbf{GSI}$)}, which can reflect the merged data distribution of sampled clients by measuring the dissimilarity between uploaded gradients. Based on this indicator, we propose a gradient similarity--aware learning rate adaptation mechanism to adaptively adjust the server learning rates.
Furthermore, our method will adjust the learning rate for each parameter group (i.e., weight matrix) individually based on its own $GSI$, in order to solve the issue of the inconsistent fluctuation patterns of $GSI$ across parameter groups and achieve more precise adjustments. Extensive experiments on multiple benchmarks show that our method consistently brings improvement to various state-of-the-art federated optimization methods in various settings.

\section{Related work}
\label{related work}

Federated Averaging (FedAvg) is first proposed by~\citet{fedavg}  for the federated learning, and its convergence property is then widely studied~\citep{fedavg_convergence, scaffold}. While FedAvg behaves well on the i.i.d.\ data with client sampling, further studies~\citep{scaffold} reveal that its performance on the non-i.i.d.\ data degrades greatly. Therefore, existing studies focus on improving the model's performance when the training samples are non-i.i.d.\ under the partial client participation setting. They can be divided into the following categories:

\noindent \textbf{Advancing the optimizer used in the server's updating:} Compared with FedAvg that is equivalent to applying SGD during the server's updating, other studies take steps further to use advanced server optimizers, such as using SGDM~\citep{fedavgm, slowmo} and adaptive optimizers~\citep{fedopt} like Adam~\citep{adam}. However, their improvement comes from the effects of using advanced server optimizers, and they do not truly target on addressing the problems of the non-i.i.d.\ data distribution and the client sampling. We show that our method can be effectively applied with these federated optimizers.

\noindent \textbf{Improving the directions of the averaged server gradients:} (1) Some studies~\citep{fednova,inverse_reweight,fedadp} manage to introduce better gradients' re-weighting procedures during server's aggregation than the naive averaging~\citep{fedavg}. However, they either require an extra global dataset for validation or fail to tackle the situation where most of the sampled clients have similar but skewed data distributions. (2) Instead of focusing on the server side, other studies choose to improve the local training on the client side. For example, FedProx~\citep{fedprox} and FedDyn~\citep{feddyn} choose to add the regularization terms to make local gradients' directions more consistent; SCAFFOLD~\citep{scaffold} and VRLSGD~\citep{VRLSGD} send the historical aggregated gradients to each client for helping correct its local gradient's direction. In this paper, we consider from an orthogonal angle that aims to find an optimal server learning rate after the averaged gradient is determined. Thus, our method can be naturally combined with them.

\noindent \textbf{Introducing new client sampling strategies:} Being aware of the important role the client sampling plays in federated learning, some studies~\citep{communication_efficient_optimal_sampling,optimal_client_sampling, power_of_choice, clustered_sampling} try to propose better client sampling schemes  than the default \textit{uniform sampling without replacement strategy}~\citep{fedavg}. They attempt to select the most important clients based on some statistics of their local gradients, such as the gradient's norm. Our work is also orthogonal to them and can bring additional improvement when applied with them.

\section{Methodology}
\label{sec: method}

\subsection{Problem definition}
\label{subsec: problem def}
The optimization target of the federated learning~\citep{fedavg} can be formalized as
\begin{equation}
    \min\limits_{\boldsymbol{\theta}} L(\boldsymbol{\theta}) =

    \min\limits_{\boldsymbol{\theta}} \sum\nolimits_{k=1}^{N}\frac{n_{k}}{n}L_{k}(\boldsymbol{\theta}),
\end{equation}
where $N$ is the total number of clients, $n_{k}$ is the number of local training samples on client $k$, $n=\sum\nolimits_{k=1}^{N}n_{k}$ is the total number of training samples, and
$L_{k}(\boldsymbol{\theta})=\frac{1}{n_{k}}\sum\nolimits_{i=1}^{n_{k}}l_{k}(\boldsymbol{\theta}; \boldsymbol{x}_{i}^{(k)}, \boldsymbol{y}_{i}^{(k)})
$
is the local training objective for client $k$.

The most popular federated learning framework is the FedAvg~\citep{fedavg}, which broadcasts the current global model $\boldsymbol{\theta}^{t}$ to the available clients when the $(t+1)$-th round begins and allows each client to perform the multi-step centralized training on its own data. Then each client only sends the accumulated local gradient $\boldsymbol{g}^{t}_{k}$ ($k=1,\cdots,N$) to the server, and the server will aggregate (average) the local gradients and update the global model as the following:
\begin{equation}
\label{eq: fedavg_alg}
\boldsymbol{\theta}^{t+1} = \boldsymbol{\theta}^{t} - \eta_{s} \sum\nolimits_{k=1}^{N}\frac{n_{k}}{n} \boldsymbol{g}_{k}^{t},
\end{equation}
where $\eta_{s}$ is the server learning rate.
The server will send the $\boldsymbol{\theta}^{t+1}$ to clients again for the next round's training.

In most real cases, due to the great number of clients and the server's limited communication capability, the server will perform a \textit{client sampling} strategy to sample a small portion of clients only in each round.
Also, since we assume the number of local updates performed during each client's local training is the same in each training round following previous studies~\citep{fedavg,fedopt},\footnote{Our later proposed method in Section~\ref{subsec: FedGLAD} can also be applied in the setting where the number of local updates varies across clients, combining with specific optimization methods like FedNova~\citep{fednova}.} then all $n_{k}$ are assumed to be the same. Therefore, Eq.~(\ref{eq: fedavg_alg}) can be re-written as
\begin{equation}
\label{eq: simplified_fedavg}
    \boldsymbol{\theta}^{t+1} = \boldsymbol{\theta}^{t} - \eta_{s} \frac{1}{r} \sum\nolimits_{k \in S(N,r;t)} \boldsymbol{g}_{k}^{t},
\end{equation}
where $r$ is the number of sampled clients, $S(N,r;t)$ represents a sampling procedure of sampling $r$ clients from $N$ clients in the $(t+1)$-th round, and we abbreviate it to $S^{t}$ in the following paper.

\subsection{Analysis of the impact of client sampling}
\label{subsec: analysis of client sampling}

As we can except, in a given round, the averaged gradient under partial client participation will deviate from the ideal gradient under full client participation, which we think has a reliable direction since it contains the data information of all clients. Thus, the motivation of most current studies can be summarized into optimizing the following problem:
\begin{equation}
\label{eq: previous optimization target}
     \max\limits_{\boldsymbol{g}^{t}} \{ \texttt{CosSim}(\boldsymbol{g}^{t}, \boldsymbol{g}^{t}_{c})\},
\end{equation}
where $\boldsymbol{g}^{t} =\frac{1}{r} \sum\nolimits_{k \in S^{t}} \boldsymbol{g}_{k}^{t}$ and $\boldsymbol{g}^{t}_{c} =\frac{1}{N} \sum\nolimits_{k=1}^{N} \boldsymbol{g}_{k}^{t}$. They achieve this goal indirectly from the perspective of improving the local gradients, either by adjusting the direction of $\boldsymbol{g}_{k}^{t}$~\citep{fedprox, scaffold, feddyn}, or by sampling better groups of clients~\citep{optimal_client_sampling, AdaFL}.

However, based on the analysis in Figure~\ref{fig: demo}, we are motivated to mitigate the inconsistent
between $\boldsymbol{g}^{t}$ and $\boldsymbol{g}^{t}_{c}$ from a different perspective, that is \textbf{scaling $\boldsymbol{g}^{t}$ to achieve the nearest distance between it and $\boldsymbol{g}^{t}_{c}$} (refer to Figure~\ref{fig: distance}). By doing so, a great advantage is to \textbf{avoid updating the model towards a skewed direction overly when the uploaded gradients are similar but skewed} (refer to the right top of Figure~\ref{fig: demo}). Then our optimization target can be written as:\footnote{In the following paper, $\| \cdot\|$ represents the Frobenius norm.}
\begin{equation}
\label{eq: optimal LR target}
    \eta^{t}_{o} = \mathop{\arg\min}\limits_{\eta^{t}} \| \eta^{t} \boldsymbol{g}^{t} - \eta_{s} \boldsymbol{g}^{t}_{c} \|^{2},
\end{equation}
where $\eta_{s}$ is the optimal global learning rate used under the full client participation setting.

Moreover, we point out that since our method is orthogonal to previous studies, we can get a more comprehensive optimization target by combining Eq.~(\ref{eq: optimal LR target}) with Eq.~(\ref{eq: previous optimization target}) as:
\begin{equation}
\label{eq: true target}
\begin{aligned}
      \min\limits_{\eta^{t}}  \|  \eta^{t} \boldsymbol{g}^{t} - \eta_{s} \boldsymbol{g}^{t}_{c} \|^{2}, \quad \text{s.t.} \quad \boldsymbol{g}^{t}  = \arg\max\limits_{\boldsymbol{g}} \{ \texttt{CosSim}(\boldsymbol{g}, \boldsymbol{g}^{t}_{c})\}.
     \end{aligned}
\end{equation}
However, in the scope of this paper, we only focus on the first part of Eq.~(\ref{eq: true target}) that is overlooked by previous studies, while \textbf{assuming $\boldsymbol{g}^{t}$ is already obtained or optimized by existing methods}.
Then, by solving Eq.~(\ref{eq: optimal LR target}) we can get
\begin{equation}
\label{eq: optimal LR solution}

    \eta^{t}_{o} = \eta_{s} \left( \langle\boldsymbol{g}^{t}, \boldsymbol{g}_{c}^{t}\rangle/ \| \boldsymbol{g}^{t}\|^{2}\right),
\end{equation}
where $<\cdot, \cdot>$ is an element-wise dot product operation.

In the $(t+1)$-th round given $\boldsymbol{\theta}^{t}$, $\boldsymbol{g}^{t}_{c}$ is the determined but unknown parameters, which means it is impossible to calculate $<\boldsymbol{g}^{t}, \boldsymbol{g}^{t}_{c}>$ precisely. However, we can still find the factors that will affect $\eta_{o}^{t}$ by analyzing its upper bound:
\begin{equation}
\label{eq: optimmal LR approx}
\begin{aligned}
    \eta^{t}_{o} & \leq \eta_{s}  \frac{  \| \boldsymbol{g}^{t}_{c} \|}{\| \boldsymbol{g}^{t} \|}
    = \frac{  \eta_{s}  \| \boldsymbol{g}^{t}_{c} \|}{\sqrt{\frac{1}{r} \sum_{k \in S_{t}}\| \boldsymbol{g}^{t}_{k} \|^{2}  }} \frac{\sqrt{\frac{1}{r} \sum_{k \in S_{t}}\| \boldsymbol{g}^{t}_{k} \|^{2}  }}{\| \boldsymbol{g}^{t} \|}

    .
    \end{aligned}
\end{equation}
In the first equality of Eq.~(\ref{eq: optimmal LR approx}), we make the further derivation to separate the \textbf{scale component} $\sqrt{\frac{1}{r} \sum_{k \in S_{t}}\| \boldsymbol{g}^{t}_{k} \|^{2}}$ that is decided by the scales of the local gradients, and the \textbf{direction component} $\| \boldsymbol{g}^{t} \| / \sqrt{\frac{1}{r} \sum_{k \in S_{t}}\| \boldsymbol{g}^{t}_{k} \|^{2}  }$ that depends on the similarity between the directions of local gradients, from $ \| \boldsymbol{g}^{t} \|$. Now in the first term of the final equation of Eq.~(\ref{eq: optimmal LR approx}), $\| \boldsymbol{g}^{t}_{c}\|$ can be considered as an unknown constant in this round given $\boldsymbol{\theta}^{t}$; the scale component $\sqrt{\frac{1}{r} \sum_{k \in S_{t}}\| \boldsymbol{g}^{t}_{k} \|^{2}}$ can also be regarded as a constant which is barely affected by the sampling results in current round, since we assume each client performs the same number of local updates, so the difference between the norms of all local gradients can be negligible (especially when the local optimizer is Adam),\footnote{We make the detailed discussion and provide the empirical evidence in Appendix~\ref{appendix: validate the equal norms of local gradients}.} compared with the difference of local gradients' directions. However, the second term $(\sqrt{\frac{1}{r} \sum_{k \in S_{t}}  \| \boldsymbol{g}^{t}_{k} \|^{2}  }) / \| \boldsymbol{g}^{t} \| $ indeed depends on the merged data distribution of currently sampled clients, as the directions of the local gradients depends on the sampled clients' local data distributions. Since it measures the normalized similarity between local gradients, we name it
the \textbf{Gradient Similarity--aware Indicator ($\mathbf{GSI}$)}, and aim to utilize it to adjust the server learning rates:
\begin{equation}
\label{eq: gsi}

    GSI^{t} = \sqrt{(\sum\nolimits_{k \in S_{t}} \| \boldsymbol{g}^{t}_{k} \|^{2} )/
    (r  \| \boldsymbol{g}^{t} \|^{2})}.
\end{equation}

The reason why we do not choose $1 / \| \boldsymbol{g}^{t} \|$ as the indicator and make the further derivation in Eq.~(\ref{eq: optimmal LR approx}) is, the scales of the gradients will naturally decreases when training goes on, and $1 / \| \boldsymbol{g}^{t} \|$ will be in an increasing trend. This leads to the consistently increasing server learning rates, which makes SGD based optimization (which happens in the server aggregation phase) fail at the end of training. However, $GSI$ is a normalized indicator whose scale is stable across rounds.

We surprisingly find that the $GSI$ happens to have the similar form to the gradient diversity concept proposed in the distributed learning field~\citep{gradient_diversity}, however, they have different functions: the gradient diversity concept is originally used to illustrate that allocating heterogeneous data to different nodes helps distributed learning; while in federated learning, the local data distributions and the sampling results in each round have been decided, but $GSI$ can reflect the reliability of the sampling results and be further used to mitigate the negative impacts of bad sampling results. Specifically, if the training data is non-i.i.d., and if the sampled clients have more similar local data distributions, which will be reflected in more similar local gradients and smaller $GSI$, the merged data distribution of sampled clients is more likely to be skewed from the global data distribution. Then the aggregated server gradient may not be fully trusted, and the server learning rate should be relatively smaller. Otherwise, the $GSI$ is larger, and the server learning rate should be larger.

\subsection{Gradient similarity--aware learning rate adaptation for federated learning}
\label{subsec: FedGLAD}

Though we can not calculate the exact $\eta^{t}_{o}$ from Eq.~(\ref{eq: optimmal LR approx}), the above analysis motivates us to use $GSI$ as an indicator to relatively adjust the current round's server learning rate.
However, the problem that $\| \boldsymbol{g}_{c}^{t}\|$ is unknown still remains. To tackle this problem, we further derive the scale component and direction component of $\|\boldsymbol{g}_{c}^{t} \|$, and extend Eq.~(\ref{eq: optimmal LR approx}) as
\begin{equation}
\label{eq: optimmal LR final}
\begin{aligned}
    \eta^{t}_{o}  \leq \eta_{s}  \frac{  \| \boldsymbol{g}^{t}_{c} \|}{\| \boldsymbol{g}^{t} \|}
   & =
     \frac{\sqrt{\frac{1}{N} \sum_{k =1}^{N}|| g_{k}^{t} ||^{2}  }}{\sqrt{\frac{1}{r} \sum_{k \in S_{t}}|| g_{k}^{t} ||^{2}  }}
\frac{ || g_{c}^{t} ||}{\sqrt{\frac{1}{N} \sum_{k =1}^{N}|| g_{k}^{t} ||^{2}  }}  \frac{\sqrt{\frac{1}{r} \sum_{k \in S_{t}}|| g_{k}^{t} ||^{2}  }}{|| g^{t} ||}
\\ &=
 \frac{\sqrt{\frac{1}{N} \sum_{k =1}^{N}|| g_{k}^{t} ||^{2}  }}{\sqrt{\frac{1}{r} \sum_{k \in S_{t}}|| g_{k}^{t} ||^{2}  }} \frac{GSI^{t}}{GSI^{t}_{c}} \approx  \frac{GSI^{t}}{GSI^{t}_{c}}
    .
    \end{aligned}
\end{equation}
We point put that, the first term $\sqrt{\frac{1}{N} \sum_{k =1}^{N}|| g_{k}^{t} ||^{2}  } / \sqrt{\frac{1}{r} \sum_{k \in S_{t}}|| g_{k}^{t} ||^{2}  }$ in Eq.~(\ref{eq: optimmal LR final}) can be regarded as an constant 1 according to our assumption on the scale component mentioned above. The second term includes $GSI^{t}_{c}$ that is the $GSI$ under the full client participation assumption and is unknown, and \textbf{we choose to estimate it with a baseline chosen as the exponential average of historical $\mathbf{GSI}$s}:
\begin{equation}
\label{eq: exp_gsi}
\begin{aligned}
    GSI_{c}^{t} \approx B^{t} = \beta B^{t-1} + (1 - \beta) GSI^{t-1}, \quad t=1,2,\cdots, \quad
    B^{0} = GSI^{0}.
\end{aligned}
\end{equation}
Therefore, the adjusted learning rate $\hat{\eta}_{t}$ can be calculated by multiplying the initial global learning rate $\eta_{0}$\footnote{$\eta_{0}$ is a hyper-parameter that can be tuned in the same way as previous methods did to get their optimal server learning rate $\eta_{s}$. Thus, this is not an extra hyper-parameter we introduced in our work.} with the ratio of $GSI^{t}$ in the current round and the estimator $B^{t}$ :
\begin{equation}
\label{eq: adjusted_lr}

    \hat{\eta}^{t} = \eta_{0} \left( GSI^{t} / B^{t} \right).
\end{equation}

\begin{figure}[t]
\begin{minipage}{.428\linewidth}
\vskip 0.1in
\begin{center}
\centerline{\includegraphics[width=1.0\linewidth]{imgs/distance_v2.pdf}}
\caption{
Illustration of our motivation.
}
\label{fig: distance}
\end{center}

\begin{center}
\centerline{\includegraphics[width=1.0\linewidth]{imgs/20news_var_term.pdf}}
\caption{Fluctuation patterns of $GSI$s in different parameter groups when federated training on 20NewsGroups~\citep{20news} with DistilBERT~\citep{distilbert}.}
\label{fig: var_term_in_different_param_group}
\end{center}

\end{minipage}
\hfil
\begin{minipage}{.558\linewidth}
\begin{center}
\makeatletter\def\@captype{table}\makeatother
\begin{tabular}{l}
\hline
\textbf{Algorithm 1}\quad Gradient Similarity--Aware Learning \\Rate Adaptation for Federated Learning\\
\hline
\textbf{Server Input:}  $\boldsymbol{\theta}^{0}$, $\eta_{0}$, $\beta=0.9$, $\gamma=0.02$\\
\textbf{for} each round $t=0$ \textbf{to} $T$ \textbf{do} \\
\quad sample client $S^{t} \subset \{1,\cdots,N\}$\\
\quad \textbf{for} $k \in S^{t}$ \textbf{in parallel do} \\
\quad \quad
 $\boldsymbol{g}_{k}^{t} \leftarrow$ {\bfseries LocalTraining}$(k,\boldsymbol{\theta}^{t}, D_{k}) $ \\
\quad \textbf{end for} \\
\quad $\boldsymbol{g}^{t} \leftarrow \frac{1}{r} \sum\nolimits_{k\in S^{t}} \boldsymbol{g}^{t}_{k}$ \\
\boxit{client}{A}{B}\quad\textbf{for}\tikzmk{A} each param. group $\boldsymbol{\theta}_{P}^{t}$'s gradient $\boldsymbol{g}^{t}_{p}$ \textbf{do}\\
\quad\quad $GSI^{t}_{P} \leftarrow \sqrt{( \sum_{k \in S_{t}} \| \boldsymbol{g}^{t}_{P,k} \|^{2} ) / ( r  \| \boldsymbol{g}^{t}_{P} \|^{2}})$ \\
\quad\quad $B^{0}_{P} \leftarrow GSI^{0}_{P}$\\
\quad\quad $\tilde{\eta}^{t}_{P} \leftarrow \min\{\max\{GSI^{t}_{P} / B^{t}_{P}, 1 - \gamma t\}, 1+\gamma t\} $ \\
\quad\quad $B^{t+1}_{P} \leftarrow \beta B^{t}_{P} + (1-\beta) GSI^{t}_{P}$ \\
\quad\quad $\boldsymbol{g}^{t}_{P} \leftarrow \tilde{\eta}^{t}_{P} \boldsymbol{g}^{t}_{P} $ \\
 \tikzmk{B} \quad \textbf{end for}\\
\quad $\boldsymbol{\theta}^{t+1} \leftarrow \boldsymbol{\theta}^{t} -  \eta_{0} \boldsymbol{g}^{t} $ \\
\textbf{end for}\\
\hline
\end{tabular}
\end{center}
\end{minipage}
\vskip -0.2in
\end{figure}

\paragraph{Parameter group--wise adaptation} Based on the above analysis, a direct solution is to regard all parameters as a whole and universally adjust the global learning rate based on Eq.~(\ref{eq: adjusted_lr}).
However, in our preliminary explorations, we visualize the fluctuation patterns of $GSI$ in different parameter groups (i.e., weight matrices) when using FedAvg to train on 20NewsGroups~\citep{20news}\footnote{The experimental settings are the same as that introduced in Section~\ref{subsec: datasets and models}. We also visualize the results on CIFAR-10~\citep{cifar} and put the results in Appendix~\ref{appendix: GSI in CIFAR-10}, while the conclusions are the same.} in Figure~\ref{fig: var_term_in_different_param_group}, and find that the $GSI$s have very different fluctuation degrees in different parameter groups whether from the same layer or not.
Thus, to achieve more precise adaptations, we give each parameter group $\boldsymbol{\theta}_{P}$ an independent adaptive learning rate based on its own $GSI$ in each round:
\begin{equation}
\label{eq: adjusted_lr_for_P}

     \hat{\eta}^{t}_{P} = \eta_{0}\left( GSI^{t}_{P} / B^{t}_{P}\right), \quad t=0,1,\cdots ,
\end{equation}
where $GSI^{t}_{P}$ and $B^{t}_{P}$ are calculated based on the averaged gradient of the $\boldsymbol{\theta}_{P}^{t}$ only from Eq.~(\ref{eq: gsi}).

\paragraph{Adaptation with dynamic bounds} Moreover, as we can see from Figure~\ref{fig: var_term_in_different_param_group}, the $GSI$ varies much more greatly at the beginning than it does when the training becomes stable. The reason is that, when the global model is newly initialized, the local gradients are not stable, leading to the great variability of $GSI$s. In addition, there are not enough historical $GSI$s for estimating $B^{t}_{P}$ at the beginning, which will further cause the $GSI_{P}^{t} / B_{P}^{t}$ and $\hat{\eta}^{t}_{P}$ to vary greatly. Therefore, to avoid doing harm to the initial training caused by the great variations of the learning rates, we set dynamic bounds which will restrict the parameter group--wise adaptive learning rates at the initial stage but gradually relax restrictions as the training becomes stable. The finally used learning rate for $\boldsymbol{\theta}_{P}^{t}$ is:
\begin{equation}
\label{eq: true_lr}
    \tilde{\eta}^{t}_{P} = \eta_{0}\min\{\max\{\hat{\eta}^{t}_{P} / \eta_{0}, 1 - \gamma t\}, 1+\gamma t\}, \quad t=0,1,\cdots ,
\end{equation}
where $\gamma$ is the bounding factor that controls the steepness of dynamic bounds. We name our method the \textit{\textbf{G}radient Similarity--Aware \textbf{L}earning R\textbf{A}te A\textbf{D}aptation for \textbf{Fed}erated learning (\textbf{FedGLAD})}. The simplified illustration is in Algorithm 1, and the detailed version is in Appendix~\ref{appendix: variations}.

Though the previous analysis is based on the FedAvg framework, our proposed algorithm can also be applied with other popular federated optimization methods (e.g., FedProx~\citep{fedprox}, FedAvgM~\citep{fedavgm}, FedAdam~\citep{fedopt}). We make a detailed discussion about these variations, and illustrate how to effectively combine FedGLAD with them in Appendix~\ref{appendix: variations}.

\section{Experiments and analysis}
\label{sec: experiments and analysis}

\subsection{Datasets, models, data partitioning and client sampling}
\label{subsec: datasets and models}
We perform experiments on two image classification tasks: CIFAR-10~\citep{cifar} and MNIST~\citep{mnist}, and two text classification tasks: 20NewsGroups~\citep{20news} and AGNews~\citep{agnews}.
The statistics of all datasets can be found in Appendix~\ref{appendix: datasets and models}. We use the ResNet-56~\citep{resnet} as the global model for CIFAR-10 and the same convolutional neural network (CNN) as that used in~\citet{fedopt} for MNIST. We use DistilBERT~\citep{distilbert} as the backbone for two text classification datasets following recent studies~\citep{scaling_FL, fednlp}. We also explore the performance of using BiLSTM~\citep{lstm} as the backbone in Appendix~\ref{appendix: bilstm}.

To simulate realistic non-i.i.d.\ data partitions, we follow previous studies~\citep{fedma, fednlp} by taking the advantage of the Dirichlet distribution $\text{Dir}(\alpha)$.
In our main experiments, we choose the non-i.i.d. degree hyper-parameter $\alpha$ as 0.1. We also conduct extra experiments on $\alpha=1.0$. The results are in Appendix~\ref{appendix: alpha=1.0}, and the conclusions remain the same.

The total number of clients for each dataset is all 100, and the number of clients sampled in each round is 10. We adopt the widely-used \textit{uniform sampling without replacement} strategy for client sampling~\citep{fedavg,  impact_client_sampling}. We further explore the impacts of using different sampling ratios and sampling strategies in Section~\ref{subsec: different sampling ratios} and Section~\ref{subsec: different sampling strategies} respectively.

\subsection{Baseline methods and training details}
\label{subsec: training details}
We choose 4 popular federated optimization methods as baselines,\footnote{We also perform experiments with SCAFFOLD~\citep{scaffold}, but we find it can not work well in our main experiments' settings. The detailed discussions can be found in Appendix~\ref{appendix: scaffold}.} which can be divided as: (1) \textbf{Using SGD as the server optimizer:} FedAvg~\citep{fedavg} and FedProx~\citep{fedprox}; (2) \textbf{Using advanced server optimizers:} FedAvgM~\citep{fedavgm, slowmo} and FedAdam~\citep{fedopt}. The detailed descriptions of them are in the Appendix~\ref{appendix: baseline methods}. We show that our method can be applied to any of them to further improve the performance.

As for the local training, we choose the client optimizers for image and text tasks as SGDM and Adam respectively.
For each dataset, we adopt the optimal local learning rate $\eta_{l}$ and batch size $B$ which are optimal in the centralized training as the local training hyper-parameters for all federated baselines. Then we allow each baseline method to tune its own best server learning rate $\eta_{s}$ on each dataset.
The number of local epochs is 5 for two image classification datasets, and 1 for two text classification datasets. We also conduct the experiments to explore the impact of the number of local epochs $E$, the results and analysis are in Appendix~\ref{appendix: different local epochs}.
The optimal training hyper-parameters (e.g., server learning rates) will be tuned based on the training loss of the previous 20\% rounds~\citep{fedopt}. Details of all the training hyper-parameters (including the total number of communication rounds for each dataset)
and further explanations are in Appendix~\ref{appendix: training details}.

When applying our method with each baseline method in each setting, \textbf{we use the original training hyper-parameters tuned before in that experiment directly without extra tuning}. In Appendix~\ref{appendix: sensitivity of bounding factor}, we explore the sensitivity to the bounding factor $\gamma$ of our method, and find $\gamma=0.02$ can be a generally good choice in all cases. Thus, in our main experiments, we fix $\gamma$ as $0.02$.
The exponential decay factor $\beta$ is fixed as $0.9$. We run each experiment on 3 different seeds, and on each seed
we save the average test accuracy (\%) over the last 10 rounds~\citep{fedopt}. Finally, we report the mean and the standard deviation of the average accuracy over the last 10 rounds on all seeds. Our code is mainly based on FedNLP~\citep{fednlp} and FedML~\citep{fedml}.

\subsection{Main results}
\label{subsec: main results}
The results of our main experiments are displayed in Table~\ref{tab: main results}. We also display the results of the centralized training on each dataset in Appendix~\ref{appendix: centralized results}. The main conclusion is, \textbf{our method can consistently bring improvement to all baselines on both image and text classification tasks in almost all settings}. Note that though the standard deviation in each experiment in Table~\ref{tab: main results} may seems a bit large compared with the improvement gap, we point out that actually \textbf{on almost each seed of one specific experiment, applying our method can gain the improvement over the corresponding baseline.} We put the result on each seed in Appendix~\ref{appendix: each seed result} for comparison.

\input{tables/main_results_new}

(1) First of all, our method effectively improves the performance of FedAvg in all settings, which directly validates our analysis in Section~\ref{subsec: FedGLAD} that it is better to set adaptive server learning rates based on the client sampling results. FedProx also uses SGD as the server optimizer but tries to improve the local gradients by making them have more consistent directions.
However, FedProx requires the carefully tuning of its hyper-parameter $\mu$ to achieve better performance than FedAvg (refer to Appendix~\ref{appendix: best mu}), and there is no guarantee that FedProx can outperform FedAvg in any setting (same conclusion for SCAFFOLD). This means that empirically adjusting the directions of local gradients may bring negative effects. Additionally, we find that FedAvg applied with FedGLAD can achieve comparable performance with FedProx in some cases.
This result indicates that \textbf{optimizing the scales of the averaged gradients by adjusting the server learning rates is equally as important as optimizing the directions of local gradients before aggregation}, while the former is overlooked by previous studies. Also, our method can further help FedProx to gain more improvement, since they are two complementary aspects for optimizing the full target in Eq.~(\ref{eq: true target}).

(2) FedAvgM and FedAdam belong to another type of baselines, which uses advanced server optimizers. For these two methods, the averaged server gradient in the current round will continuously affect the training in the future rounds since there is a momentum term to store it. Our method aims to enlarge the scales of the relatively reliable averaged gradients, while down-scaling the averaged gradients that deviate greatly from the global optimization direction in specific rounds. Thus, \textbf{our method can improve the learning when server optimizers include a momentum term to stabilize training}.

(3) When comparing the results in Table~\ref{tab: main results} with the results on $\alpha=1.0$ in Table~\ref{tab: main results alpha=1.0} in Appendix~\ref{appendix: alpha=1.0}, we can get that on each dataset, the trend is \textbf{the improvement brought by our method is greater if the non-i.i.d.\ degree is larger (i.e., smaller $\mathbf{\alpha}$)}. Our explanation is when the label distribution is more skewed, the impact of client sampling becomes larger. More analysis is in Appendix~\ref{appendix: alpha=1.0}.

\subsection{The effect of FedGLAD under extremely bad client sampling scenarios}

\begin{wrapfigure}{R}{0.5\linewidth}
\begin{center}
\vskip -0.1in
\centerline{\includegraphics[width=1.0\linewidth]{imgs/mnist_E=5_v2.pdf}}
\caption{The good effect of FedGLAD on stabilizing training when extremely bad client sampling scenarios happen.}
\label{fig: mnist extremely bad case}
\end{center}
\end{wrapfigure}

In Figure~\ref{fig: demo}, we draw an example under the extremely bad client sampling scenario to illustrate the necessity of adapting server learning rates according to the client sampling results. Here, we validate this motivation by conducting a special experiment on MNIST with FedAvg. Specifically, while keeping other settings unchanged, we make sure that clients sampled in the 40-th round only have almost one same category of samples. Figure~\ref{fig: mnist extremely bad case} displays the curves of the evaluation accuracy and the $GSI$ calculated by regarding all parameters as a whole after each round.

We notice that in the 40-th round, the accuracy of FedAvg drops greatly due to the bad client sampling results, and the corresponding $GSI$ is also the smallest (i.e., close to 1) during the training. After applying FedGLAD, the accuracy degradation problem is effectively mitigated, and the accuracy curve is much more stable. The indicates that, \textbf{our method can help to stabilize the training when extremely bad client sampling scenarios happen}.

\section{Deep explorations}
\label{sec: more analysis}
In this section, we take steps further to make more explorations of our method. We choose FedAvg and FedAvgM as baseline methods, and perform experiments on CIFAR-10 and 20NewsGroups (with the backbone DistilBERT) datasets with the non-i.i.d.\ degree $\alpha=0.1$. The bounding factor $\gamma$ of our method is fixed as $0.02$.
All experiments are run on 3 random seeds.

\subsection{The effect of adjusting learning rates individually for different parameter groups}
\label{subsec: the effect of param group-wise}

Here, we explore the benefits of using parameter group--wise adaptation compared with the direct solution that considers all parameters as a whole and adjusts the server learning rate universally (\textbf{universal adaptation}, mentioned at the beginning of Section~\ref{subsec: FedGLAD}). The results are in Figure~\ref{fig: effect of param group-wise}.

Firstly, the universal adaptation strategy can already bring improvement to the baselines without adaptation. This indicates that dynamically adjusting the server learning rates based on the dissimilarity between local gradients can indeed help federated learning. Moreover, the performance of parameter group--wise adaptation strategy is consistently better than the universal adaptation strategy, which validates our motivation based on the findings in Figure~\ref{fig: var_term_in_different_param_group} that the non-i.i.d.\ data distribution has different impacts on the different parameter groups, so we should deal with them individually.

\begin{figure}[t]
\begin{center}
\subfigure[Results on CIFAR-10.]{
\begin{minipage}[t]{0.48\linewidth}  \centerline{\includegraphics[width=1\linewidth]{imgs/cifar10_effect_of_group_wise_E=5.pdf}}
\end{minipage}
}
\subfigure[Results on 20NewsGroups.]{
\begin{minipage}[t]{0.48\linewidth}
\centerline{\includegraphics[width=1\linewidth]{imgs/20news_effect_of_group_wise.pdf}}
\end{minipage}
}
\caption{Comparisons between two learning rate adaptation strategies of FedGLAD.
\label{fig: effect of param group-wise}
\end{center}
\vskip -0.2in
\end{figure}

\subsection{The impact of using different sampling ratios}
\label{subsec: different sampling ratios}
In the main experiments, we fix the sampling ratio as $0.1$. Here, we explore our method's performance in the settings where 5, 10, 20 out of 100 clients are sampled in each round. The optimal server learning rates for baselines are re-tuned here. The results are in Table~\ref{tab: different sampling ratios}. The overall trend is \textbf{the improvement brought by our method is more significant when the sampling ratio is smaller}. That is because when the number of sampled clients increases, the merged data distribution of sampled clients is more likely to be consistent with the merged data distribution of all clients, thus the impact of client sampling decreases.

\begin{figure}[t]

\begin{minipage}{.48\linewidth}
\centering
\makeatletter\def\@captype{table}\makeatother
\caption{Results under different sampling ratios.}
\label{tab: different sampling ratios}
\vskip 0.15in
\scalebox{0.77}{
\begin{tabular}{llccc}
\toprule
\multirow{2.5}{*}{Dataset} & \multirow{2.5}{*}{Method} & \multicolumn{3}{c}{Sampling Ratio}  \\
\cmidrule(lr){3-5}
 & & 5\% & 10\% & 20\%  \\
\midrule
\multirow{4.5}{*}{CIFAR-10} & FedAvg &  29.91 & 35.30  & 52.05 \\
& ~~ \small{+ FedGLAD} & \textbf{33.35} & \textbf{38.44} & \textbf{53.40} \\
\cmidrule{2-5}
 & FedAvgM & 34.72  & 48.79 & 63.56  \\
 & ~~ \small{+ FedGLAD} & \textbf{36.48} & \textbf{50.91}  & \textbf{63.84} \\
\cmidrule{1-5}
\multirow{4.5}{*}{20NewsGroups} & FedAvg & 63.09 & 72.84  & 76.31 \\
& ~~ \small{+ FedGLAD} & \textbf{63.49} & \textbf{73.14}  & \textbf{76.35} \\
\cmidrule{2-5}
 & FedAvgM & 61.51 & 75.44  & \textbf{80.08} \\
 & ~~ \small{+ FedGLAD} & \textbf{62.48} & \textbf{75.91}  & 79.98 \\
\bottomrule
\end{tabular}
}
\vskip -0.1in
\end{minipage}
\hfil
\begin{minipage}{.485\linewidth}
\makeatletter\def\@captype{table}\makeatother \centering
\caption{Results of different sampling strategies.}
\label{tab: various sampling strategies}
\vskip 0.15in
\scalebox{0.77}{
\begin{tabular}{llcc}
\toprule
\multirow{2.5}{*}{Dataset} & \multirow{2.5}{*}{Method} & \multicolumn{2}{c}{Sampling Strategy}  \\
\cmidrule(lr){3-4}
 & & MD & AdaFL  \\
\midrule
\multirow{4.5}{*}{CIFAR-10} & FedAvg & 41.44 & 39.85 \\
& ~~ \small{+ FedGLAD} & \textbf{42.20} & \textbf{41.39}  \\
\cmidrule{2-4}
 & FedAvgM & 51.77 & 52.76 \\
 & ~~ \small{+ FedGLAD} & \textbf{53.85} & \textbf{53.68} \\
 \cmidrule{1-4}
\multirow{4.5}{*}{20NewsGroups} & FedAvg & 73.36 & 72.14 \\
& ~~ \small{+ FedGLAD} & \textbf{74.04} & \textbf{72.92} \\
\cmidrule{2-4}
 & FedAvgM & 75.22 &  76.18 \\
 & ~~ \small{+ FedGLAD} & \textbf{75.60} & \textbf{76.40} \\
\bottomrule
\end{tabular}
}
\vskip -0.1in
\end{minipage}

\end{figure}

\subsection{Combining with various client sampling strategies}
\label{subsec: different sampling strategies}
In this subsection, we want to show that our method can also be combined with other sampling mechanisms besides the normal one used in our main experiments. We conduct experiments with two additional sampling strategies: (1) \textbf{Multinomial Distribution--based Sampling (MD)}~\citep{impact_client_sampling}: considering sampled clients as $r$ i.i.d.\ samples from a Multinomial Distribution; (2) \textbf{Dynamic Attention-based Sampling (AdaFL)}~\citep{AdaFL}: dynamically adjusting the sampling weights of clients during the training. The sampling ratio is 0.1. The results in Table~\ref{tab: various sampling strategies} show that our method can also work well with these sampling strategies, since they aim to sample better groups of clients in each round while our method improves the learning after the sampling results are determined.

\section{Conclusion}

In this paper, we study and mitigate the difficulty of federated learning on non-i.i.d.\ data caused by the client sampling practice. We point out that the negative client sampling will make the aggregated gradients unreliable, thus the server learning rates should be dynamically adjusted according to the situations of the local data distributions of sampled clients. We then theoretically find that the optimal server learning rate in each round is positively related to an indicator, which can reflect the merged data distribution of sampled clients in that round. Based on this indicator, we design a novel learning rate adaptation mechanism to mitigate the negative impact of client sampling. Extensive experiments validate the effectiveness of our method.

\section*{Acknowledgments}
This work was supported by a Tencent Research Grant. Xu Sun is the corresponding author of this paper.

\newpage

redpajama_set_name: RedPajamaArXiv


It's time to think about getting up – notice we said 'think about getting up'. Now's the time in that warm glow between sleepfulness and wakefulness when you can bathe in the cool refreshing music we play and the gentle thoughts we bring.
The idea is to slowly move you into the day. As dawn rises, so will you, refreshed and ready to 'seize the day' with both hands.


text:
Youngsters tackle emergency at sea
TWO ferries trying to avoid a small yacht in Harwich Harbour ended up smashing into each other, tipping containers full of hazardous materials into the water.
That was the scene facing more than 100 students as they got to grips with the emergency - a challenge to test their ability to deal with a live and developing situation and get different organisations to work together to tackle the incident.
The horrific collision was purely imaginary, but to the youngsters taking part it was a real situation and they had to work out how to deal with it with the help of experts from shipping, port and other marine industries.
The students - aged 14 and 15, from nine schools, including Felixstowe, Ipswich and east and south Suffolk - were told a Norfolk Lines ro-ro ferry and a China Shipping container vessel had collided as they tried to avoid a meandering yacht, the Suffolk Daydreamer. More than 40 containers were lost overboard, including dangerous cargo.
They were then put into teams, mixing schools together, to work out what to do.
Felixstowe port corporate affairs officer Rachael Jackson said during the day new information about the changing situation in the harbour was given to the groups, presenting them with fresh challenges to overcome.
"It was purposefully a very challenging scenario, and a very realistic one, too," she said.
One of the aims of the event - organised by the Port of Felixstowe and Suffolk Education Business Partnership, in conjunction with the Haven Gateway Partnership, at Kesgrave Community Centre - was to encourage the youngsters to think about careers in the shipping and logistics industry.
Chris Lewis, chief executive officer of Hutchison Ports (UK) Limited, which owns the port, said: "It is vital that the ports industry does all it can to attract young people."
Those taking part included Felixstowe port's emergency services division, marine and port services, and corporate affairs department; Harwich Haven Authority, Johnson Stevens Agencies, Norfolk Line, Maritime and Coastguard Agency, RNLI, and Suffolk County Council emergency planning services.

redpajama_set_name: RedPajamaCommonCrawl


Lucas Films has commissioned some of the finest artists and illustrators to recreate those classic scenes and moments that made Star Wars such an internationally acclaimed success. Many of the pieces are hand signed by the artists and a special highlight is a collection of pieces from Ralph McQuarrie, famed artist and illustrator who designed many of the Star Wars characters including Darth Vader and Chewbacca.
In many ways Star Wars kicked off the Pop Culture Phenomenon. Due to its overwhelming success Hollywood adapted to this new ethos of movie making by writing more action packed, special effects movies. Without Star Wars you wouldn't have Avatar.
Star Wars is the third highest grossing film series in history, behind only Harry Potter and the James Bond Films. As of 2008 it had earned $4.41 billion at the box office. From the first movie's release on May 25, 1977 to the final film released in 2005, Silver K will have the all your favourite moments on display. With over 150 pieces in the exhibition including such celebrated characters as Yoda, Luke Skywalker, Princess Leia, R2-D2, Darth Vader, C-3PO and many, many more it will be an exhibition not to miss.
The show will be an intergalactic journey through the creative world of Star Wars.
The Art of Star Wars opens November 26th.


text:
zombies without zombies
In I Am Legend there's the perfectly and wonderfully accurate representation of flooded tunnels in an electricity-free, and thus pump-free post-apocalyptic New York:
Thanks to this image, I can now cobble together my Five Obstructions-style article about zombie movies that doesn't give more than a passing mention of zombies. It's the infrastructure that matters in zombie flicks.
Tags I Am Legend (2007)
Tom Tykwer, establishing shot master
Tom Tykwer movies have really great establishing shots (they also have gorgeous and effective overhead shots, something I'll write about later). I want to base this claim in a quick look at the mostly unexceptional The International, for which John Mahiffe, in addition to a mess of location-specific ADs, was the second unit director.
Perhaps the key to the establishing shots in The International is the way in which they make clear how people use the space on screen. One of the first images in the film
approximates a point-of-view shot from the seat of a car in the car park that takes up the bottom third of the image. It's not exactly that point of view – it's from slightly higher up – but it's close enough to make the space and its use clear. That is to say, while there's certainly an aestheticization going on – The International does its fair share of action-flick tourism – there's also a sense of a world working outside the bounds of the film's narrative. This sense that the world is more than a stage for the film's conspiracy is quite clear in the Milan sequences. Take, for example, the investigation of Umberto Calvini's assassination:
The wonderful thing about this shot is that the sequence that follows it seems to do almost nothing with all the information provided in the shot. Instead, Clive Owen's Louis Salinger and Naomi Watts's Eleanor Whitman look at the bullet holes and their local police ally chats to the other police on the site. The actual/official investigation occurs on the edge of the frame, untouched by the film. The film's paranoid narrative offers one explanation – any investigation will be whitewash – but film language offers another. When Salinger and Whitman notice the trajectories don't match, they take to the neighbouring building's roof to see this:
Here Tykwer not only visualizes the "alternative point of view" that sees trajectories and connections that the on-site investigators can't/won't see, but it also shows the actual investigation site in a wide, long shot. There's a world at work outside and beyond the film's protagonists that isn't concerned with playing antagonist to them. There are more things in heaven and earth than dreamed of in our cinema, after all.
A non-generic treatment of a generic situation bears out my claim. Salinger, on foot, pursues The Real Shooter, who is in a car. But when Salinger gets to the traffic snarl where the Shooter is trapped, he faces too many choices:
In this establishing shot (of sorts) we get a sense of the life of the city. But we're still in the realm of genre. Our gun-toting hero must now go from car to car, scaring the shit out of people who have nothing to do with the chase (that is, the film's narrative) other than proximity. But that doesn't happen in The International. Instead, as Salinger weaves through the cars, Tykwer goes to another kind of shot he loves, the overhead.
No one gets menaced. No one flees amid squealing tyres. In fact, the cars pull away at a reasonable speed, as if the light changed and they did what any driver would do. There's a world out there and The International's global financial chicanery just doesn't enter into things that much. Salinger's left alone, confused and the world keeps turning.
All in all, Tykwer gets more ideological mileage out of The International's establishing shots than most films get out of their entire cinematic vocabulary, even in a middling effort.
Tags The International (2009)
the worse it is the better
Barring some multimillion dollar windfall for UC, I won't be teaching in the College of Arts next year. They've already cut travel funding for permanent staff, but as contract staff I didn't have access to that fund anyway. But there's a teaching-adaptations conference in Tasmania in February. I hope to schedule some away-from-the-uni time to check out some of the most incredible (usually green) architecture around. It's also the home of the great fictional Aussie Rules player Geoff Hayward. I might even spot a Tasmanian wolf and make some coin. So I sent out an abstract about how I teach She's the Man,which is perhaps the worst movie I have taught more than once. Note: The CINE 101 students who argue for Vigil are wrong. On to the abstract.
"The worse it is the better: On teaching sub-par film adaptations"
One of the problems with teaching film adaptations of canonical works of literature is that such adaptations are often used to illustrate the literature rather than to study film in its own right. The canon may have expanded in our lifetime, but it still consists of a limited number of texts with a presumed shared greatness – and our students know this. My paper outlines an approach that asks students to locate the rules that underwrite the canon. Assigning "bad" adaptations is one way to help students query canon formation by demanding a definition and defense of the canon.
I argue that we can teach "bad" adaptations "backwards": the lecturer argues for the greatness of the "bad" film over, for example, Shakespeare, forcing students to articulate exactly what makes Shakespeare worthy of canonization. Such an approach locates the process of canon formation in the classroom and places the responsibility in student hands.
I use the 2006 Twelfth Night adaptation She's the Man as my example. She's the Man will never be accused of committing greatness to film. Teaching it seems like masochism at best, and sadism at worst. Yet, as my paper shows, teaching a "bad" adaptation makes the questions – and answers – about contemporary political and cultural life that adapting canonical works makes possible much more readily accessible. That is to say, if we reimagine She's the Man's many failures as virtues, and Twelfth Night's many virtues as failures,we can not only articulate the boundaries and functions of the canon, but we can also address the problems and politics of canon formation in a way that affirms the role that literature plays in cultural life.
Tags She's the Man (2006)
Categories cinema studies, literary criticism, teaching
research is important
It's not every movie about academics that gets what the life is like. Take, for example, the truly execrable The Mirror Has Two Faces. I'm probably more guilty of redirecting my dreams of being a stand-up comic in the George Carlin–Richard Pryor vein into my teaching style than most lecturers working in New Zealand. But even I find Barbara Streisand's Rose Morgan impossible to stomach when it comes to her thoughts about how to teach or how to make people care about your research. It's such a smug, self-satisfied idea to place at the centre of the film that, especially when you have Jeff Bridges' near-boundless charm and Lauren Bacall's gloriously withering disdain for others waiting around for a better movie to emerge.
However, when I went to The Bourne Legacy a while back with my friend Steve, I was shocked to hear a couple of tossed-off lines of dialog that captured the academic mind-set. After Aaron and Marta have escaped from the make-it-look-like-a-suicide set piece, Aaron starts going a little bonkers about his blues and greens. At this point, Steve leaned over and said to me, in trailer voice, "the thrilling adventures of a junkie in search of his next fix!" But then Weisz's Marta puts Aaron in his place (I paraphrase): "You don't know what I gave up for this! I couldn't publish. I couldn't conference. I couldn't even talk about my research." It's that special recipe of well-founded pride in intellectual prowess and pursuits with a healthy glop of pettiness that sums up The Academic.
Tags The Bourne Legacy (2012), The Mirror Has Two Faces (1996)
Categories cinema studies, teaching


text:
Barlumi
Great gallerists
New meetings
Let's go to the museum
Art to eat
Dubito Ergo Cogito
Instarte
Looking around
Maybe not everyone knows…
Special Weekend
Great Italy
La posta dell'avvocato
Venice Renaissance
Black Number
AW books
Painting to reset
Cesare Orler
Naples • Lawrence Carroll is on stage at Madre
On display 80 works created during his 30-year career
The Madre museum is hosting the first major Lawrence Carroll retrospective since his death in 2019. On display until the 5th of September are 80 works created during the Melbourne-based artist's 3-decade career.
Carroll moved to New York in 1984 to work as a graphic designer and illustrator and began experimenting with painting by interpreting it in his own way. He immediately proves to be an unusual painter who isn't part of any specific movement. He meets the protagonists of pop art, minimal art and new dada and manages to capture their essence by veering toward a reset painting.
IN THE EXHIBIT EACH WORK RECALLS
THE ONE NEXT TO IT AND SUGGESTS NEW CONNECTIONS.
One could say that Lawrence Carroll is white, just as Klein is blue and Kapoor is black. In the exhibition curated by Gianfranco Maraniello we perceive the austerity and at the same time the candor of a style that investigates the very tools of painting. Had he been born in Italy we would have called him an "analytical painter," but as the citizen of the world that he is, we leave it to him to define his work. "In my painting the forms are always changing, their very placement in space changes, this restlessness of thought and spirit is what in a way makes my works alive and keeps them moving." Carroll looks up to artists who anticipated him
in similar pursuits. Objects, media, compositional strategies and formal organizations on the one hand are not theoretical novelties; on the other hand, they demonstrate his ability to metabolize and reformulate twentieth-century languages to the point of emerging with his own identity.
"THIS RESTLESSNESS OF THOUGHT AND SPIRIT IS WHAT
IN A WAY MAKES MY WORKS ALIVE AND KEEPS THEM MOVING"
One can sense in the use of raw fabrics the echo of Alberto Burri and Jannis Kounellis and that of Cy Twombly and Robert Rauschenberg in the use of white and sign. The arrangement does not follow a thematic or chronological order, but an elective one, which Aby Warburg would have called "of good neighborliness. "Each work recalls the one next to it, and the environment suggests new connections. In first place we find the works and the emotional relationship with the viewer. Maraniello says, "Putting together a Lawrence Carroll exhibition today means remaining faithful to his restlessness, corresponding to the vitality of works that continue to search for their place to live by questioning the unending possibilities of painting."
Lawrence Carroll
Museo Madre
Curated by Gianfranco Maraniello
Until the 5/09
Madre - museo d'arte contemporanea Donnaregina, Napoli/Naples Ph: Amedeo Benestante Courtesy Madre - museo d'arte contemporanea Donnaregina
Cesare Orler firmly believes in the equivalence of art and life and would like to turn his life into a work of art, to paraphrase D'annunzio. He has a degree in Conservation of Cultural Heritage and Performing Arts Management, which he took in Venice, and is completing the master's degree Programme in Contemporary Art History. He manages "Cesare's Corner", a TV broadcast on OrlerTV whose aim is to disseminate Contemporary Art. He closely follows emerging Italian artists and curates exhibitions and critical texts on them. He is a keen supporter of AW ArtMag. In addition to art, he also likes cinema and drinking beer, of which he is a refined connoisseur. Perhaps of all these things he can do well only the last one.
Abramović at Castel dell'Ovo: the ecstasy of Marina
Maria Angela Tiozzi - November 19, 2020
London • Abakanowicz major exhibition at Tate Modern
Alessandro Romanini - January 26, 2023
BEAUTY CARE - New York. Nick Cave's retrospective at the Guggenheim
Francesco Agnellini - January 20, 2023
Vertigo - Last Judgement by Marison Ray
Luciano Caprile - January 15, 2023
When Steel becomes Art - Venice, Great Alviani exhibition at Scuola grande della misericordia
Cesare Orler - January 13, 2023
SIAE artist? No article
Lorella Pagnucco Salvemini - December 31, 2022
Coco and Pablo masters of style
Michela Bassanello - December 7, 2022
Art in the sanctuary
Elena Altemura - December 5, 2022
I destroy, therefore I create
Elena Altemura - November 21, 2022
Francesco Agnellini - November 10, 2022
Venice: a meeting with Karole P. B. Vail, Peggy Guggenheim's granddaughter
Myriam Zerbi - November 17, 2020
Banksy's legal problems: anonymous artist? No rights
Cesare Orler - November 18, 2020
Olafur Eliasson at the Guggenheim: like the northen lights
Elena Altemura - March 2, 2021
The "SURREAL" life of Salvador Dalì
Alessio Musella - November 23, 2020
Art without frontiers
Toti Carpentieri - May 5, 2022
A Conversation with Mario Botta: transforming nature into culture
Giuseppe Calandriello - March 9, 2021
That great and transgressive visionary
Toti Carpentieri - July 22, 2021
© AW Artmag 2023. All rights reserved.