text:
\section{Introduction} \label{sec:prob-setup}
\begin{comment}
{\tiny
\begin{itemize}
\item Importance of FL
\item[\xmark] Importance of HPO in FL
\item General problem of HPO in FL
\item Scope of the problem considered in this paper
  \begin{itemize}
  \item Only global HPs for models; no aggregator HPs
  \item no personalization; all parties use same HP and eventually use the same model
  \item Targeting a single FL-training to address communication overhead
  \item No ``weight-sharing'' kind of trick available
  \end{itemize}
\item Motivation for selected scope
  \begin{itemize}
  \item Individual trainings already communication heavy;
  \item Communication overhead is one of the critical challenges in FL; ``weaker'' parties affected most when using techniques robust to ``stragglers''
  \item HPO usually requires many-many trainings, making potential communication overhead impractical, adversely affecting weakest parties the most
  \item Single-shot HPO focuses on one end of the communication overhead spectrum, trying to have minimal communication overhead due to HPO
  \item Not the only useful setup to consider, but a critical point in the spectrum that needs explicit attention
  \end{itemize}
\item Contribution
  \begin{itemize}
  \item Novel scheme for single-shot HPO in FL setting
  \item Empirical evaluation on various OpenML datasets~\citep{OpenML2013}
  \end{itemize}
  \item[\xmark] outline of the paper
\end{itemize}
}
\end{comment}

Traditional machine learning (ML) approaches require training data to be gathered at a central location where the learning algorithm runs. In real world scenarios, however, training data is often subject to privacy or regulatory constraints restricting the way data can be shared, used and transmitted. Examples of such regulations include the European General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), Cybersecurity Law of China (CLA) and HIPAA, among others.
Federated learning (FL), first proposed in~\citet{mcmahan2017communication}, has recently become a popular approach to address privacy concerns by allowing collaborative training of ML models among multiple parties where each party can keep its data private.

\paragraph{{\sf FL-HPO}\xspace problem.} Despite the privacy protection FL brings along, there are many open problems in FL domain~\citep{kairouz2019advances,khodak2021federated}, one of which is hyper-parameter optimization for FL.
Existing FL systems require a user (or all participating parties) to pre-set (agree on) multiple hyper-parameters (HPs) (i)~for the model being trained (such as number of layers and batch size for neural networks or tree depth and number of trees in tree ensembles), and (ii)~for the the aggregator (if such hyper-parameters exist).  Hyper-parameter optimization (HPO) for FL is important because the choice of HPs can have dramatic impact on performance \citep{mcmahan2017communication}.

While HPO has been widely studied in the centralized ML setting, it comes with unique challenges in the FL setting.  First, existing HPO techniques for centralized training often make use of the entire dataset, which is not available in FL. Secondly, they train a vast variety of models for a large number of HP configurations which would be prohibitively expensive in terms of communication and training time in FL settings. Thirdly, one important challenge that has not been adequately explored in FL literature is support for tabular data, which are widely used in enterprise settings \citep{ludwig2020ibm}.
One of the best models for this setting is based on gradient boosting tree algorithms~\citep{friedman2001greedy} which are different from the stochastic gradient descent algorithm used for neural networks.  Recently, a few approaches have been proposed for {\sf FL-HPO}\xspace,
however they focus on handling HPO using personalization techniques~\citep{khodak2021federated} and neural networks~\citep{khodak2020weight}.
To the best of our knowledge, there is no HPO approach for FL systems to train non-neural network models, such as XGBoost~\citep{xgboost} that is particularly common in the enterprise setting.
\paragraph{Scope.}
In this paper, we address the aforementioned challenges of {\sf FL-HPO}\xspace.
We focus on the problem where the model HPs are shared across all parties
and we seek a set of HPs and train a single model that is eventually used by all parties for testing/deployment.
Moreover, we impose three further requirements that make the problem more challenging:
(\cond{1})~we {\em do not make any assumption} that two models with different HPs can perform some form of ``weight-sharing'' (which is a common technique used in various HPO and neural architecture search (NAS) schemes for neural networks to reduce the computational overhead of HPO and NAS), allowing our solution to be applied beyond neural networks~\citep{khodak2020weight}.
(\cond{2})~we seek to {\bf perform ``single-shot'' {\sf FL-HPO}\xspace}, where we have {\em limited} resources (in the form of computation and communication overhead) which allow training only a single model via federated learning (that is, a single HP configuration), and
(\cond{3})~we {\em do not assume that parties have independent and identically distributed (IID) data distributions}.
\paragraph{Contributions.}
Given the above {\sf FL-HPO}\xspace problem setting, we make the following contributions:
\begin{itemize}[noitemsep,topsep=0pt]
\item (\S \ref{sec:method}) We present a novel framework {\em {\bf F}ederated {\bf Lo}ss Su{\bf R}face {\bf A}ggregation} ({\sf FLoRA}\xspace) that leverages meta-learning techniques to utilize {\bf local and asynchronous} HPO on each party to perform single-shot HPO for the global {\sf FL-HPO}\xspace problem.
\item (\S \ref{sec:og_analysis}) We provide theoretical guarantees for the set of HPs selected by {\sf FLoRA}\xspace covering both IID and Non-IID cases. To the best of our knowledge, this is the first rigorous theoretical analysis for {\sf FL-HPO}\xspace problem and also the first optimality gap constructed in terms of the estimated loss given a target distribution.
\item (\S \ref{sec:emp}) We evaluate {\sf FLoRA}\xspace on the {\sf FL-HPO}\xspace of Gradient Boosted Decision Trees~(GBDTs), Support Vector Machines~(SVMs) and Multi-layered Perceptrons~(MLPs) on seven classification datasets from OpenML~\citep{OpenML2013}, highlighting (i) its performance relative to the baseline, (ii) the effect of various choices in this scheme, and (iii) the effect of the number of parties on the performance.
\end{itemize}

\section{Related work}\label{sec:related}
\begin{comment}
{\tiny
\begin{itemize}
\item FL review
\item FL HPO: FedEx paper~\citep{khodak2020weight,khodak2021federated}
\item[\xmark] (Maybe) FL XGBoost~\citep{li2020practical,ong2020adaptive}
\item HPO review: general HPO, meta-learning~\citep{vanschoren2018meta}
  \begin{itemize}
  \item Why some usual HPO schemes (like BO or multi-fidelity Hyperband) are not directly appropriate for {\sf FL-HPO}\xspace
  \item Why meta-learning is an useful direction when thinking of single-shot HPO
  \item Highlight that usually meta-learning helps with (and use for) ``few-shot'' problems; not ``single-shot''
  \end{itemize}
\end{itemize}
}
\end{comment}
\paragraph{Performance optimization of FL systems.} One of the main challenges in FL is achieving high accuracy and low communication overhead. FedAvg~\citep{pmlr-v54-mcmahan17a} is a predominant algorithm used for training in FL and several optimization schemes build on it. It is executed in multiple global rounds. At each round, the clients perform stochastic gradient descent (SGD) updates on their parameters based on their local objective functions. They subsequently send their updates to the server, which averages them and transmits their mean back to the clients. Several approaches have been devised for optimizing the communication performance of FL systems. Initially, communication optimizations included performing multiple SGD local iterations at the clients and randomly selecting a small subset of the clients to compute and send updates to the server~\citep{pmlr-v54-mcmahan17a}. Subsequently, compression techniques were used to minimize the size of model updates to the server. It has been shown that the accuracy and communication performance of these techniques depend highly on their HPs~\citep{pmlr-v54-mcmahan17a}.
\paragraph{{\sf FL-HPO}\xspace approaches.}
Recent optimization approaches adapt HPs such as the local learning rate at each client~\citep{Koskela19Learning,Mostafa19Robust,Reddi20Adaptive}, the number of local SGD iterations (which affect the frequency of server updates)~\citep{wang2019adaptive}.
In~\citet{Dai20FBO,Dai21Differentially}, Dai et.al. address Federated Bayesian Optimization. Although using HPO with multiple HPs, the problem setup is quite different than Federated Learning: they focus on a single party using information from other parties to accelerate its own Bayesian Optimization, instead of building a model for all parties. Federated Network Architecture Search (FNAS) approaches search for architectural HPs of deep learning CNN models by running locally NAS algorithms and then aggregating the NAS architecture weights and model weights using FedAvg~\citep{He20toward,Garg20direct,Xu20federated}. These approaches have shown empirical gains but lack theoretical analysis.
Inspired from the NAS technique of weight-sharing,~\citep{khodak2020weight, khodak2021federated} proposed FedEx, a {\sf FL-HPO}\xspace framework to accelerate a general HPO procedure, i.e., successive halving algorithm (SHA), for many SGD-based FL algorithms. Fedex focuses on building personalized models for parties by tuning local HPs of the parties. They provide a theoretical analysis for a special case of tuning a single HP (learning rate) in a convex optimization setting.

Our framework improves on the above approaches in several ways. {\bf 1) It is more general}, as it can tune multiple HPs and is applicable to non SGD-training settings such as gradient boosting trees. This is achieved by treating {\sf FL-HPO}\xspace as a black-box HPO problem, which has been addressed in centralized HPO literature using grid search, random search~\citep{bergstra2012random} and Bayesian Optimization approaches~\citep{shahriari2016taking}.  The key challenge is the requirement to perform computationally intensive evaluations on a large number of HPO configurations, where each evaluation involves training a model and scoring it on a validation dataset. In the distributed FL setting  this problem is exacerbated because validation sets are local to the parties and each FL training/scoring evaluation is communication intensive.  Therefore a brute force application of centralized black-box HPO approaches that select hyper-parameters in an outer loop and proceed with FL training evaluations is not feasible.
{2) \bf It yields minimal HPO communication overhead.} This is achieved by building a loss surface from local asynchronous HPO at the parties that yields a single optimized HP configuration used to train a global model with a single FL training. {\bf 3) It is the first that theoretically characterizes optimality gap in an {\sf FL-HPO}\xspace setting}, for the case we focus in this paper (creating a global model by tuning multiple global HPs).

\section{Methodology} \label{sec:method}
\begin{comment}
{\tiny
\begin{itemize}
\item Formal {\sf FL-HPO}\xspace problem + notation\todo{Can the problem formulation itself be posed as a contribution?}
\item Local HPOs; send loss-HP pairs to aggregator
\item Aggregator builds global loss surface
\item Choices for loss surfaces + intuition of what each loss surface is doing
\item Aggregator picks single HP
\item FL with selected HP initiated with all involved parties
\end{itemize}
}
\end{comment}

In the centralized ML setting, we would consider a model class $\mathcal{M}$ and its corresponding learning algorithm $\mathcal{A}$ parameterized collectively with HPs $\boldsymbol{\theta} \in \boldsymbol{\Theta}$, and given a training set $D$, we can learn a single model $\mathcal{A}(\mathcal{M}, \boldsymbol{\theta}, D) \to m \in \mathcal{M}$. Given some predictive loss $\mathcal{L}(m, D')$ of any model $m$ scored on some holdout set $D'$, the centralized HPO problem can be stated as
\begin{equation}
\label{eq:central-HPO}
\min\nolimits_{\boldsymbol{\theta} \in \boldsymbol{\Theta}} \mathcal{L}(\mathcal{A}(\mathcal{M}, \boldsymbol{\theta}, D), D').
\end{equation}
In the most general FL setting, we have $p$ parties $P_1, \dots, P_p$ each with their private local training dataset $D_i, i \in [p] = \{1, 2, \ldots, p\}$. Let $D = \cup_{i=1}^p D_i$ denote the aggregated training dataset and $\overline{D} = \{D_i\}_{i \in [p]}$ denote the set of per-party datasets. Each model class (and corresponding learning algorithm) is parameterized by global HPs $\boldsymbol{\theta}_G \in \boldsymbol{\Theta}_G$ shared by all parties and per-party local HPs $\boldsymbol{\theta}_L^{(i)} \in \boldsymbol{\Theta}_L, i \in [p]$ with $\boldsymbol{\Theta} = \boldsymbol{\Theta}_G \times \boldsymbol{\Theta}_L$. FL systems usually include an aggregator with its own set of HPs $\boldsymbol{\phi} \in \boldsymbol{\Phi}$. Finally, we would have a FL algorithm
\begin{equation}\label{eq:def-fl-alg}
    \mathcal{F} \left( \mathcal{M}, \boldsymbol{\phi}, \boldsymbol{\theta}_G, \{ \boldsymbol{\theta}_L^{(i)} \}_{i \in [p]}, \mathcal{A}, \overline{D} \right) \to m \in \mathcal{M},
\end{equation}
which takes as input all the relevant HPs and per-party datasets and generates a model. in this case, the {\sf FL-HPO}\xspace problem can be stated in the two following ways depending on the desired goals:
(i)~Ideally, for a global holdout dataset $D'$ (a.k.a validation set, possibly from the same distribution as the aggregated dataset $D$), the target problem is:
\begin{equation}
\label{eq:fl-hpo-1}
\min_{\boldsymbol{\phi} \in \boldsymbol{\Phi}, \boldsymbol{\theta}_G \in \boldsymbol{\Theta}_G, \boldsymbol{\theta}_L^{(i)} \in \boldsymbol{\Theta}_L, i \in [p]}
\mathcal{L}\left( \mathcal{F} \left(
\mathcal{M}, \boldsymbol{\phi}, \boldsymbol{\theta}_G, \{ \boldsymbol{\theta}_L^{(i)} \}_{i \in [p]}, \mathcal{A}, \overline{D}
\right), D' \right).
\end{equation}
\begin{comment}
\begin{equation}
\label{eq:fl-hpo-1}
\min_{\boldsymbol{\phi} \in \boldsymbol{\Phi}, \boldsymbol{\theta}_G \in \boldsymbol{\Theta}_G, \boldsymbol{\theta}_L^{(i)} \in \boldsymbol{\Theta}_L, i \in [p]}
\mathcal{L}\left( \mathcal{F}, D' \right).
\end{equation}
\end{comment}
(ii)~An alternative target problem would involve per-party holdout datasets $D_i', i \in [p]$ as follows:
\begin{equation}
\label{eq:fl-hpo-2}
\min_{\boldsymbol{\phi} \in \boldsymbol{\Phi}, \boldsymbol{\theta}_G \in \boldsymbol{\Theta}_G, \boldsymbol{\theta}_L^{(i)} \in \boldsymbol{\Theta}_L, i \in [p]}
\mathsf{Agg}\left(
\left\{
  \mathcal{L}\left( \mathcal{F} \left(
  \mathcal{M}, \boldsymbol{\phi}, \boldsymbol{\theta}_G, \{ \boldsymbol{\theta}_L^{(i)} \}_{i \in [p]}, \mathcal{A}, \overline{D}
  \right), D'_i \right),
  i \in [p] \right\}
\right),
\end{equation}
\begin{comment}
\begin{equation}
\label{eq:fl-hpo-2}
\min_{\boldsymbol{\phi} \in \boldsymbol{\Phi}, \boldsymbol{\theta}_G \in \boldsymbol{\Theta}_G, \boldsymbol{\theta}_L^{(i)} \in \boldsymbol{\Theta}_L, i \in [p]}
\mathsf{Agg}\left(
\left\{
  \mathcal{L}\left( \mathcal{F}, D'_i \right),
  i \in [p] \right\}
\right),
\end{equation}
\end{comment}
where $\mathsf{Agg}: \mathbb{R}^p \to \mathbb{R}$ is some aggregation function (such as average or maximum) that scalarizes the $p$ per-party predictive losses.

Contrasting problem \eqref{eq:central-HPO} to problems \eqref{eq:fl-hpo-1} \& \eqref{eq:fl-hpo-2}, we can see that the {\sf FL-HPO}\xspace is significantly more complicated than the centralized HPO problem. In the ensuing presentation, we focus on problem~\eqref{eq:fl-hpo-1} although our proposed single-shot {\sf FL-HPO}\xspace scheme can be applied and evaluated for problem~\eqref{eq:fl-hpo-2}. We simplify the {\sf FL-HPO}\xspace problem in the following ways: (i)~we assume that there is no personalization so there are no per-party local HPs $\boldsymbol{\theta}_L^{(i)}, i \in [p]$, (ii)~we only focus on the model class HPs $\boldsymbol{\theta}_G$, deferring HPO for aggregator HPs $\boldsymbol{\phi}$ for future work, and (iii)~we assume there is a global holdout/validation set $D'$ which is only used to evaluate the final global model's performance but {\em can not be accessed} during HPO process.
Hence the problem we will study is stated as for a fixed aggregator HP $\boldsymbol{\phi}$:
\begin{equation} \label{eq:fl-hpo-1a}
\min\nolimits_{\boldsymbol{\theta}_G \in \boldsymbol{\Theta}_G}
\mathcal{L}\left( \mathcal{F} \left(
\mathcal{M}, \boldsymbol{\phi}, \boldsymbol{\theta}_G, \mathcal{A}, \overline{D}
\right), D' \right).
\end{equation}
This problem appears similar to the centralized HPO problem~\eqref{eq:central-HPO}. However, note that the main challenges in \eqref{eq:fl-hpo-1a} is (i)~the need for a federated training for each set of HPs $\boldsymbol{\theta}_G$, and (ii)~the need to evaluate the trained model on the global validation set $D'$ (which is usually not available in usual {\sf FL-HPO}\xspace setting). Hence it is not practical (from a communication overhead and functional  perspective) to apply existing off-the-shelf HPO schemes to problem~\eqref{eq:fl-hpo-1a}. In the subsequent discussion, for simplicity purposes, we will use $\boldsymbol{\theta}$ to denote the global HPs, dropping the ``$G$'' subscript.

\subsection{Leveraging local HPOs}
\begin{algorithm}[t]
\caption{Single-shot {\sf FL-HPO}\xspace with Federated Loss Surface Aggregation ({\sf FLoRA}\xspace)}
\label{alg:fl-hpo-lsa}
\begin{algorithmic}[1]
\STATE {\bf Input:}{$\boldsymbol{\Theta}, \mathcal{M}, \mathcal{A}, \mathcal{F}, \{ (D_i, D_i') \}_{i \in [p]}, T$}
  \FOR {each party $P_i, i \in [p]$}
    \STATE Run HPO to generate $T$ (HP, loss) pairs%
    \begin{equation}\label{eq:local-hpo-data}

    E^{(i)} = \left \{
    (\boldsymbol{\theta}_t^{(i)}, \mathcal{L}_t^{(i)}), t \in [T]
    \right\},
    \end{equation}%
    where $\boldsymbol{\theta}_t^{(i)} \in \boldsymbol{\Theta},
    \mathcal{L}_t^{(i)} := \mathcal{L}(\mathcal{A}(\mathcal{M}, \boldsymbol{\theta}_t^{(i)}, D_i), D_i')$.
  \ENDFOR
  \STATE Collect all $E = \{E^{(i)}, i \in [p] \}$ in aggregator
  \STATE Generate a unified loss surface $\widehat \ell:\boldsymbol{\Theta} \to \mathbb{R}$ using $E$

  \STATE Select best HP candidate
  \begin{align}\label{eq:def:flora-thetas}
      \widehat \boldsymbol{\theta}^\star \gets \arg \min\limits_{\boldsymbol{\theta} \in \boldsymbol{\Theta}} \widehat \ell(\boldsymbol{\theta}).
  \end{align}

  \STATE Invoke federated training $m \gets \mathcal{F}(\mathcal{M}, \widehat\boldsymbol{\theta}^\star, \mathcal{A}, \overline{D})$
  \STATE {\bf Output:} FL model $m$.
\end{algorithmic}
\end{algorithm}
While it is impractical to apply off-the-shelf HPO solvers (such as Bayesian Optimization (BO)~\citep{shahriari2016taking}, Hyperopt~\citep{bergstra2011algorithms}, SMAC~\citep{hutter2011sequential}, and such), we wish to understand how we can leverage local and asynchronous HPOs in each of the parties. We begin with a simple but intuitive hypothesis underlying various meta-learning schemes for HPO~\citep{vanschoren2018meta,wistuba2018scalable}: {\em if a HP configuration $\boldsymbol{\theta}$ has good performance for all parties independently, then $\boldsymbol{\theta}$ is a strong candidate for federated training}.

With this hypothesis, we present our proposed algorithm {\bf {\sf FLoRA}\xspace} in Algorithm~\ref{alg:fl-hpo-lsa}. In this scheme, we allow each party to perform HPO locally and asynchronously with some adaptive HPO scheme such as BO (line 3). Then, at each party $i \in [p]$, we collect all the attempted $T$ HPs $\boldsymbol{\theta}_t^{(i)}, t \in [T] = \{1, 2, \ldots, T\}$ and their corresponding predictive loss $\mathcal{L}_t^{(i)}$ into a set $E^{(i)}$ (line 3, equation~\eqref{eq:local-hpo-data}). Then these per-party sets of (HP, loss) pairs $E^{(i)}$ are collected at the aggregator (line 5). This operation has at most $O(pT)$ communication overhead (note that the number of HPs are usually much smaller than the number of columns or number of rows in the per-party datasets). These sets are then used to generate an aggregated loss surface $\widehat \ell: \boldsymbol{\Theta} \to \mathbb{R}$ (line 6) which will then be used to make the final single-shot HP recommendation $\widehat \boldsymbol{\theta}^\star \in \boldsymbol{\Theta}$ (line 7) for the federated training to create the final model $m \in \mathcal{M}$ (line 8). We will discuss the generation of the aggregated loss surface in detail in \S \ref{sec:method:lsa}. Before that, we briefly want to discuss the motivation behind some of our choices in Algorithm~\ref{alg:fl-hpo-lsa}.
\paragraph{Why adaptive HPO?}
The reason to use adaptive HPO schemes instead of non-adaptive schemes such as random search or grid search is that this allows us to efficiently approximate the local loss surface more accurately (and with more certainty) in regions of the HP space where the local performance is favorable instead of trying to approximate the loss surface well over the complete HP space. This has advantages both in terms of computational efficiency and loss surface approximation.
\paragraph{Why asynchronous HPO?}
Each party executes HPO asynchronously, without coordination with HPO results from other parties or with the aggregator. This is in line with our objective to minimize communication overhead. Although there could be strategies that involve coordination between parties, they could involve many rounds of communication. Our experimental results show that this approach is effective for the datasets we evaluated for.

\subsection{Loss surface aggregation} \label{sec:method:lsa}
Given the sets of (HP, loss) pairs $E^{(i)} = (\boldsymbol{\theta}_t^{(i)}, \mathcal{L}_t^{(i)}), i \in [p], t \in [T]$ at the aggregator, we wish to construct a loss surface $\widehat \ell : \boldsymbol{\Theta} \to \mathbb{R}$ that best emulates the (relative) performance loss $\widehat \ell(\boldsymbol{\theta})$ we would observe when training the model on $\overline{D}$. Based on our hypothesis, we want the loss surface to be such that it would have a relatively low $\widehat \ell(\boldsymbol{\theta})$ if $\boldsymbol{\theta}$ has a low loss for all parties simultaneously. However, because of the asynchronous and adaptive nature of the local HPOs, for any HP $\boldsymbol{\theta} \in \boldsymbol{\Theta}$, we would not have the corresponding losses from all the parties. For that reason, we will model the loss surfaces using regressors that try to map any HP to their corresponding loss. In the following, we present four ways of constructing such loss surfaces:
\paragraph{Single global model (SGM).}
We merge all the sets $ E= \cup_{i\in [p]} E^{(i)}$ and use it as a training set for a regressor $f: \boldsymbol{\Theta} \to \mathbb{R}$, which considers the HPs $\boldsymbol{\theta} \in \boldsymbol{\Theta}$ as the covariates and the corresponding loss as the dependent variable. For example, we can train a random forest regressor~\citep{breiman2001random} on this training set $E$. Then we can define the loss surface $\widehat \ell(\boldsymbol{\theta}) := f(\boldsymbol{\theta})$. While this loss surface is simple to obtain, it may not be able to handle Non-iid party data distribution well: it is actually overly optimistic -- under the assumption that every party generates unique HPs during the local HPO, this single global loss surface would assign a low loss to any HP $\boldsymbol{\theta}$ which has a low loss at any one of the parties. This implies that this loss surface would end up recommending HPs that have low loss in just one of the parties, but not necessarily on all parties.
\paragraph{Single global model with uncertainty (SGM+U).}
Given the merged set $E= \cup_{i\in [p]} E^{(i)}$, we can train a regressor that provides uncertainty quantification around its predictions (such as Gaussian Process Regressor~\citep{williams2006gaussian}) as $f: \boldsymbol{\Theta} \to \mathbb{R}, u: \boldsymbol{\Theta} \to \mathbb{R}_+$, where $f(\boldsymbol{\theta})$ is the mean prediction of the model at $\boldsymbol{\theta} \in \boldsymbol{\Theta}$ while $u(\boldsymbol{\theta})$ quantifies the uncertainty around this prediction $f(\boldsymbol{\theta})$. We define the loss surface as $\widehat \ell(\boldsymbol{\theta}) := f(\boldsymbol{\theta}) + \alpha \cdot u(\boldsymbol{\theta})$ for some $\alpha > 0$. This loss surface does prefer HPs that have a low loss even in just one of the parties, but it penalizes a HP if the model estimates high uncertainty around this HP. Usually, a high uncertainty around a HP would be either because the training set $E$ does not have many samples around this HP (implying that many parties did not view the region containing this HP as one with low loss), or because there are multiple samples in the region around this HP but parties do not collectively agree that this is a promising region for HPs. Hence this makes SGM+U more desirable than SGM, giving us a loss surface that estimates low loss for HPs that are simultaneously thought to be promising to multiple parties.
\paragraph{Maximum of per-party local models (MPLM).}
Instead of a single global model on the merged set $E$, we can instead train a regressor $f^{(i)}: \boldsymbol{\Theta} \to \mathbb{R}, i \in [p]$ with each of the per-party set $E^{(i)}$. Given this, we can construct the loss surface as $\widehat \ell(\boldsymbol{\theta}) := \max_{i \in [p]} f^{(i)}(\boldsymbol{\theta})$. This can be seen as a much more pessimistic loss surface, assigning a low loss to a HP only if it has a low loss estimate across all parties.
\paragraph{Average of per-party local models (APLM).}
A less pessimistic version of MPLM would be to construct the loss surface as the average of the per-party regressors $f^{(i)}, i \in [p]$ instead of the maximum, defined as $\widehat \ell(\boldsymbol{\theta}) := \nicefrac{1}{p} \sum_{i=1}^p f^{(i)}(\boldsymbol{\theta})$. This is also less optimistic than SGM since it will assign a low loss for a HP only if its average across all per-party regressors is low, which implies that all parties observed a relatively low loss around this HP.

Intuitively, we believe that loss surfaces such as SGM+U or APLM would be the most promising while the extremely optimistic and pessimistic  SGM and MPLM respectively would be relatively less promising, with MPLM being superior to SGM. In the following section, we theoretically quantify the performance guarantees for MPLM and APLM, and in \S \ref{sec:emp},  we evaluate all these loss surface empirically in the single-shot {\sf FL-HPO}\xspace setting.

\section{Optimality analysis}\label{sec:og_analysis}
In this section, we provide a rigorous analysis of the sub-optimality of the HP selected by {\sf FLoRA}\xspace.
Let us first define some notation we will use throughout this section.
\begin{definition}[Loss functions]
For a given set of parties' data
$\overline{D} = \{D_i\}_{i \in [p]}$ and any $\boldsymbol{\theta} \in \boldsymbol{\Theta}$, the true target loss (any predictive performance metric, such as, the training loss) can be expressed as:
\begin{equation}\label{eq:def:loss}
\ell(\boldsymbol{\theta}, \mathcal{D}) :=
 \underbrace{\mathbb{E}_{(x, y) \sim \mathcal{D}}}_{\text{test perf. of trained model}}
\mathcal{L}(\underbrace{\mathcal{A}(\boldsymbol{\theta}, \overline D)}_{\text{trained model}}, (x, y)).
\end{equation}
Here $\mathcal{D}$ is the data distribution of the test points. Let $\tilde \ell(\boldsymbol{\theta}, \mathcal{D})$ be an estimate of the loss defined in \eqref{eq:def:loss} given some validation (holdout) set $D'$ sampled from $\mathcal{D}$, which is the model performance metric during evaluation and/or inference time.
\end{definition}
We assume the parties' training sets are collected before the federated learning such that $\overline D$ is fixed and unchanged during the HPO and FL processes, in order words, we do not consider streaming data setting.

Now we are ready to provide a more general definition of the unified loss surface constructed by {\sf FLoRA}\xspace as follows:
\begin{definition}[Unified loss surface]
Given the local loss surfaces $\widehat \ell_i: \boldsymbol{\Theta} \to \mathbb{R}$ for each party $i \in [p]$ generated by $T$ (HP, loss) pairs
$\{(\boldsymbol{\theta}^{(i)}_t, \mathcal{L}_t^{(i)}) \}_{t\in[T]}$, we can define the global loss surface $\widehat \ell : \boldsymbol{\Theta} \to \mathbb{R}$ as
\begin{equation}
    \label{eq:def:global-loss}
    \widehat \ell(\boldsymbol{\theta}) = \textstyle{\sum}_{i=1}^p \alpha_i(\boldsymbol{\theta}) \cdot \widehat \ell_i(\boldsymbol{\theta}), \alpha_i(\boldsymbol{\theta}) \in [0, 1], \textstyle{\sum}_{i=1}^{p} \alpha_i(\boldsymbol{\theta}) = 1.
\end{equation}
In particular,
\begin{itemize}[noitemsep,topsep=0pt]
    \item[i)] If $\alpha_i(\boldsymbol{\theta}) = \nicefrac{1}{p}, \ \forall i \in [p], \boldsymbol{\theta} \in \boldsymbol{\Theta}$, then this reduces to APLM loss surface.
    \item[ii)] If $\alpha_i(\boldsymbol{\theta}) = \mathbb{I} \left (\widehat\ell_i(\boldsymbol{\theta}) = \max_{j \in [p]} \widehat\ell_j(\boldsymbol{\theta}) \right)$, then this reduces to the MPLM loss surface (assuming all $\widehat\ell_j(\boldsymbol{\theta})$s are unique).
\end{itemize}
\end{definition}

We formalize the distance metric used in our analysis to evaluate the distance between two given data distributions.
\begin{definition}[1-Wasserstein distance~\citep{villani2003topics}]
For two distributions $\mu, \nu$ with bounded support, the 1-Wasserstein distance is defined as
\begin{equation} \label{eq:def:1wd}
    \mathcal W_1(\mu, \nu) := \sup_{f \in \mathsf F_1}
    \mathbb{E}_{x \sim \mu} f(x) - \mathbb{E}_{x \sim \nu} f(x),
\end{equation}
where $\mathsf F_1 = \{ f: f \text{ is continuous}, \textsf{Lipschitz}(f) \leq 1 \}$.
\end{definition}

To facilitate our analysis later, we make the following Lipschitzness assumptions regarding the loss function $\tilde \ell$ and also the per-party loss surface $\widehat \ell_i$.
\begin{assumption}[Lipschitzness]
For a fixed data distribution $\mathcal D$ and $\forall \boldsymbol{\theta}, \boldsymbol{\theta}' \in \bar{\boldsymbol{\Theta}} \subset \boldsymbol{\Theta}$, we have
\begin{align}
    | \tilde \ell(\boldsymbol{\theta}, \mathcal D) - \tilde \ell(\boldsymbol{\theta}', \mathcal D) |  &\leq \tilde L(\mathcal D) \cdot d(\boldsymbol{\theta}, \boldsymbol{\theta}'), \label{eq:def:theta-lip-tilde}\\
    | \widehat \ell_i(\boldsymbol{\theta}) - \widehat \ell_i(\boldsymbol{\theta}') |  &\leq \widehat L_i \cdot d(\boldsymbol{\theta}, \boldsymbol{\theta}'), \label{eq:def:theta-lip-hat}
\end{align}
where $d(\cdot, \cdot)$ is a certain distance metric defined over the hyper-parameter search space, see Appendix~\ref{app:def-dis-theta-space} for one definition.
For a fixed set of hyper-parameters $\boldsymbol{\theta}  \in \bar{\boldsymbol{\Theta}} \subset \boldsymbol{\Theta}$ and some data distributions $\mathcal D$ and $\mathcal D'$, we have
\begin{equation}\label{eq:def:dist-lip-hat}
    | \tilde \ell(\boldsymbol{\theta}, \mathcal D) - \tilde \ell(\boldsymbol{\theta}, \mathcal D') |  \leq \tilde \beta(\boldsymbol{\theta}) \cdot \mathcal W_1(\mathcal D, \mathcal D').
\end{equation}
\end{assumption}
\paragraph{Remark.}
Note that we explicitly use a $\bar\boldsymbol{\Theta} \subset \boldsymbol{\Theta}$ to highlight that we need Lipschitz-ness only in particular parts of the HP space. In fact, our analysis only requires the Lipschitz-ness at $\widehat \boldsymbol{\theta}^*$, the optimal HP and a HP space containing these two HPs and the set of HP tried in local HPO runs, i.e., $\boldsymbol{\theta}_t^{(i)}$, which most of the time does not cover the entire HP search space.
Moreover, the above Lipschitzness assumption w.r.t. a general HP space, which could be a combination of continuous and discrete variables, may be strong. We also show in Appendix~\ref{app:ct-theta-space} and \ref{app:alt-proof-prop-bnd-eps-i} that it can be relaxed to a milder assumption based on the modulus of continuity without significantly affecting our main results.
For simplicity, we can always assume that
$\tilde L(\mathcal D)  \leq \tilde L, \ \forall \mathcal D$ and
$\tilde \beta(\boldsymbol{\theta}) \leq \tilde \beta, \ \forall \boldsymbol{\theta}$.

Recall that the HP $\widehat\boldsymbol{\theta}^\star $ selected by {\sf FLoRA}\xspace is defined as in \eqref{eq:def:flora-thetas}.
We then define the optimal HP given by the estimated loss function for a desired data distribution $\mathcal{D}$ we want to learn as,
\begin{equation}\label{eq:def:global-theta}
\boldsymbol{\theta}^\star \in \arg \min_{\boldsymbol{\theta} \in \boldsymbol{\Theta}} \tilde \ell(\boldsymbol{\theta}, \mathcal{D}).
\end{equation}

We are interested in providing a bound for the following {\em optimality gap}:
\begin{equation} \label{eq:def:og}
    \tilde \ell(\widehat \boldsymbol{\theta}^\star, \mathcal{D}) - \tilde \ell(\boldsymbol{\theta}^\star, \mathcal{D}).
\end{equation}
Note that this bound is the optimality gap for the output of {\sf FLoRA}\xspace in terms of the estimated loss $\tilde \ell$. We state our main results in the following theorem.
Informally speaking we show how to bound the optimality gap by picking the `worst-case' HP setting that maximizes the combination of Wasserstein distances of the local data distributions and actual quality of local HPO approximation across parties.

\begin{theorem}\label{thm:og-bnd}
Consider the optimality gap defined in \eqref{eq:def:og}, where $\widehat \boldsymbol{\theta}^*$ is selected by {\sf FLoRA}\xspace with each party $i\in [p]$ collecting $T$ (HP, loss) pairs $\{(\boldsymbol{\theta}_t^{(i)}, \mathcal{L}_t^{(i)})\}_{t\in [T]}$  during the local HPO run.
For a desired data distribution $ \mathcal{D} = \sum_{i=1}^p w_i \mathcal{D}_i$, where $\{\mathcal{D}_i\}_{i\in [p]}$ are the sets of parties' local data distributions and $w_i\in [0, 1], \forall i\in [p]$, we have
\begin{align}\label{eq:og-bnd}
     &\tilde \ell(\widehat \boldsymbol{\theta}^\star, \mathcal{D}) - \tilde \ell(\boldsymbol{\theta}^\star, \mathcal{D})\nonumber\\
     & \ \le 2 \max_{\boldsymbol{\theta} \in \bar\boldsymbol{\Theta}}\textstyle{\sum}_{i\in [p]} \alpha_i(\boldsymbol{\theta})\left\{
     \tilde \beta(\boldsymbol{\theta})
     \textstyle{\sum}_{j \in [p], j \not= i} w_j \mathcal W_1 (\mathcal{D}_j, \mathcal{D}_i)

     + \left( \tilde L(\mathcal{D}_i) + \widehat L_i \right)
   \min_{t \in [T]} d(\boldsymbol{\theta}, \boldsymbol{\theta}_t^{(i)})
  + \delta_i\right\},
\end{align}
where $\delta_i$ is the maximum per sample training error for the local loss surface $\widehat \ell_i$, i.e., $\delta_i = \max_t | \mathcal{L}_t^{(i)} - \widehat \ell_i (\boldsymbol{\theta}_t^{(i)}) |$.
In particular, when all parties have i.i.d. local data distributions, \eqref{eq:og-bnd} reduces to
\begin{align*}
     & \tilde \ell(\widehat \boldsymbol{\theta}^\star, \mathcal{D}) - \tilde \ell(\boldsymbol{\theta}^\star, \mathcal{D})

     \le 2 \max_{\boldsymbol{\theta} \in \bar\boldsymbol{\Theta}}\sum_{i=1}^{p} \alpha_i(\boldsymbol{\theta}) \left\{\left( \tilde L(\mathcal{D}_i) + \widehat L_i \right) \min\limits_{t \in [T]} d(\boldsymbol{\theta}, \boldsymbol{\theta}_t^{(i)})
  + \delta_i\right\}.
\end{align*}
\end{theorem}
We make some observations regarding the above results. Firstly, the first term in our bound characterizes the errors incurred by the differences among parties' local data distributions, i.e., the magnitude of Non-IIDness in a FL system. In particular, we can see it vanish under the IID setting.
Secondly, the last two terms measure the quality of the local HPO approximation, which can be reduced if a good loss surface is selected.
Thirdly, $\min_{t \in [T]} d(\boldsymbol{\theta}, \boldsymbol{\theta}_t^{(i)})$
indicates that the optimality gap depends only on the HP trials $\boldsymbol{\theta}_t^{(i)}$ that is closest to the optimal HP setting.
Finally, if we assume each party's training dataset $D_i$ is of size $n_i$ sampled as $D_i \sim \mathcal{D}_i^{n_i}$, we can view $w_i = \tfrac{n_i}{n}$ where $n = \sum_{i=1}^p n_i$, i.e., with probability $w_i$ the desired data distribution $\mathcal{D}$ is sampled from $\mathcal{D}_i$.

In order to obtain the result in \eqref{eq:og-bnd}, we first analyze \eqref{eq:def:og} in Proposition~\ref{prop:opt-bnd}, see its proof in Appendix~\ref{app:proofs}.
Note that the local loss surfaces $\widehat \ell_i, \ i\in[p]$ are computed at a certain test/validation set sampled from the parties local data distribution $\mathcal{D}_i$. We quantify the relationship between $\widehat \ell_i(\boldsymbol{\theta})$ and the estimated loss function $\tilde \ell(\boldsymbol{\theta}, \mathcal{D}_i)$ as follows:
\begin{equation}
    \label{eq:def:local-loss}
    | \widehat \ell_i(\boldsymbol{\theta}) - \tilde \ell(\boldsymbol{\theta}, \mathcal{D}_i) | := \epsilon_i(\boldsymbol{\theta}, T).
\end{equation}

\begin{proposition}\label{prop:opt-bnd}
Consider $\widehat \boldsymbol{\theta}^\star$ and $\boldsymbol{\theta}^\star$ are two sets of HP defined in \eqref{eq:def:flora-thetas} and \eqref{eq:def:global-theta}, respectively, and $\{\mathcal{D}_i\}_{i\in [p]}$ and $\mathcal{D}$ are the sets of parties' local data distributions and the target (global) data distribution we want to learn, for a given HP space such that $\widehat \boldsymbol{\theta}^\star, \boldsymbol{\theta}^\star \in \bar\boldsymbol{\Theta} \subset \boldsymbol{\Theta}$, we have
\begin{align}
    &\tilde \ell(\widehat \boldsymbol{\theta}^\star, \mathcal{D}) - \tilde \ell(\boldsymbol{\theta}^\star, \mathcal{D})
    \label{eq:def:opt-bnd}

    \le 2 \max_{\boldsymbol{\theta} \in \bar\boldsymbol{\Theta}}\textstyle{\sum}_{i\in [p]} \alpha_i(\boldsymbol{\theta}) \left\{\tilde \beta(\boldsymbol{\theta}) \mathcal W_1(\mathcal{D}, \mathcal{D}_i) +  \epsilon_i(\boldsymbol{\theta}, T)\right\}.

\end{align}
\end{proposition}

We now dive into each term in \eqref{eq:def:opt-bnd} to provide tight bounds for $\mathcal W_1(\mathcal{D}, \mathcal{D}_i)$ and $\epsilon_i(\boldsymbol{\theta}, T)$ in the following propositions. All the proofs can be found in Appendix~\ref{app:proofs}.

\begin{proposition}\label{prop:bnd-wd}
Consider 1-Wasserstein distance we defined in \eqref{eq:def:1wd}, for a local data distribution $\mathcal{D}_i$ of any party $i, \ i\in [p]$, and $ \mathcal{D} = \sum_{i=1}^p w_i \mathcal{D}_i$ for some $w_i\in [0,1], \forall i\in [p]$, we have
\begin{equation}\label{eq:bnd-wd}
    \mathcal W_1(\mathcal{D}, \mathcal{D}_i) \leq \textstyle{\sum}_{j \in [p], j \not= i} w_j \mathcal W_1 (\mathcal{D}_j, \mathcal{D}_i).
\end{equation}
In particular, when $\mathcal{D}_i, \ i\in [p]$ are i.i.d. data distribution, i.e., all parties in a federated learning system possess i.i.d. local data distribution -- that is, $\mathcal W_1(\mathcal{D}_j, \mathcal{D}_i) = 0 \forall i,j \in [p]$ -- then  $\textstyle{\sum}_{j\in [p], j \not= i} w_j \mathcal W_1 (\mathcal{D}_j, \mathcal{D}_i) = 0$. Therefore, $\mathcal W_1(\mathcal{D}, \mathcal{D}_i) = 0, \ \forall i\in [p]$.
\end{proposition}

\begin{proposition}\label{prop:bnd-eps-i}
For any party $i, \ i\in[p]$, consider a (HP, loss) pair $(\boldsymbol{\theta}_t^{(i)}, \mathcal{L}_t^{(i)})$ collected during the local HPO run for party $i$, for any $\boldsymbol{\theta} \in \bar{\boldsymbol{\Theta}} \subset \boldsymbol{\Theta}$, we have
\begin{equation}\label{eq:bnd-eps-i}
\epsilon_i(\boldsymbol{\theta}, T)\leq
   \left( \tilde L(\mathcal{D}_i) + \widehat L_i \right)
   \min_{t \in [T]} d(\boldsymbol{\theta}, \boldsymbol{\theta}_t^{(i)})
  + \delta_i,
\end{equation}
where $\delta_i = \max_t | \mathcal{L}_t^{(i)} - \widehat \ell_i (\boldsymbol{\theta}_t^{(i)}) |$
is the maximum per sample training error for the local loss surface $\widehat \ell_i$.
\end{proposition}

Note that if we use non-parametric regression models as the loss surfaces (such as Gaussian Processes, Random Forests, etc), the per-sample training error can be made arbitrarily small (that is $\delta_i \approx 0$), but at the cost of increasing $\widehat L_i$ for $\widehat \ell_i$.

\section{Empirical evaluation} \label{sec:emp}
\begin{comment}
{\tiny
\begin{itemize}
\item What is good single-shot baseline? Motivate sklearn default as a good baseline?
\item Discuss selection of datasets
\item Evaluation metric
\item Present results;
\item discuss results
\end{itemize}
}
\end{comment}

\begin{table*}[t]
\centering
\caption{Comparison of different loss surfaces (the 4 rightmost columns)  for {\sf FLoRA}\xspace relative to the  baseline for single-shot 3-party {\sf FL-HPO}\xspace in terms of the {\em relative regret} (lower is better).
}
{\scriptsize
\begin{tabular}{lcccccc}
\toprule
Aggregate & ML Method & SGM & SGM+U & MPLM & APLM \\
\midrule
Regret &  HGB  &   [0.30, 0.47, 0.68]  &   [0.27, 0.54, 0.64]  &   [0.25, 0.43, 0.67]  &   [0.25, 0.50, 0.65] \\
Inter-quartile range &  SVM  &   [0.04, 0.38, 1.11]  &   [0.04, 0.48, 1.07]  &   [0.38, 0.91, 2.41]  &   [0.23, 0.54, 0.76] \\
&  MLP  &   [0.36, 0.80, 0.97]  &   [0.48, 0.99, 1.01]  &   [0.47, 0.89, 1.00]  &   [0.46, 0.79, 0.95] \\
& Overall & [{\bf 0.22}, {\bf 0.53}, 0.97] & [0.32, 0.55, 1.01] & [0.36, 0.61, 0.99] & [0.36, 0.57, {\bf 0.79}] \\
\midrule
{\sf FLoRA}\xspace & HGB &  6/0/1  &   6/0/1  &   7/0/0  &   7/0/0 \\
Wins/Ties/Losses & SVM &  4/0/2  &   4/0/2  &   3/0/3  &   5/0/1 \\
& MLP &   6/0/1  &   4/1/2  &   5/1/1  &   6/0/1 \\
& Overall & 16/0/4 & 14/1/5 & 15/1/4 & {\bf 18/0/2} \\
\midrule
Wilcoxon Signed-Rank Test &  HGB & (26, 0.02126)  &   (27, 0.01400)  &   (28, 0.00898)  &   (28, 0.00898) \\
1-sided & SVM  &   (18, 0.05793)  &   (17, 0.08648)  &   (9, 0.62342)  &   (15, 0.17272) \\
(statistic, p-value) & MLP  &  (21, 0.11836)  &   (15, 0.17272)  &   (18, 0.05793)  &   (24, 0.04548) \\
& Overall & (174, 0.00499) & (164, 0.00272) & (141, 0.03206) & {\bf (183.5, 0.00169)}
 \\
\bottomrule
\end{tabular}
}
\label{tab:baseline-comp}
\end{table*}

In this section, we evaluate our proposed scheme {\sf FLoRA}\xspace with different loss surfaces for the {\sf FL-HPO}\xspace on a variety of ML models -- histograms based gradient boosted (HGB) decision trees~\citep{friedman2001greedy}, Support Vector Machines (SVM) with RBF kernel and multi-layered perceptrons (MLP) (using their respective {\tt scikit-learn} implementation~\citep{scikit-learn}) on OpenML~\citep{OpenML2013} classification problems.
The precise HP search space is described in Appendix~\ref{asec:emp:search-space}.
First, we fix the number of parties $p = 3$ and compare {\sf FLoRA}\xspace to a baseline on $7$ datasets. Then we study the effect of increasing the number of parties from $p = 3$ up to $p = 100$ on the performance of our proposed scheme on $3$ datasets. The data is randomly split across parties.
We also evaluate {\sf FLoRA}\xspace with different parameter choices, in particular, the number of local HPO rounds and the communication overhead in the aggregation of the per-party (HP, loss) pairs.
Finally, we evaluate {\sf FLoRA}\xspace in a real FL testbed IBM FL~\citep{ludwig2020ibm} using its default HP setting as a baseline.
More experimental results can be found in Appendix~\ref{app:exp}.
\paragraph{Single-shot baseline.}
To appropriately evaluate our proposed single-shot {\sf FL-HPO}\xspace scheme, we need to select a meaningful single-shot baseline. For this, we choose the default HP configuration of {\tt scikit-learn}
as the single-shot baseline for two main reasons: (i)~the default HP configuration in {\tt scikit-learn} is set manually based on expert prior knowledge and extensive empirical evaluation, and
(ii)~these are also used as the defaults in the
Auto-Sklearn package~\citep{feurer2015efficient,feurer-arxiv20a}, one of the leading open-source AutoML python packages, which maintains a carefully selected portfolio of default configurations.
\paragraph{Dataset selection.}
For our evaluation of single-shot HPO, we consider
$7$ binary classification datasets of varying sizes and characteristics from OpenML~\citep{OpenML2013} such that there is at least a significant room for improvement over the single-shot baseline performance. We consider datasets which have at least $> 3\%$ potential improvement in balanced accuracy for gradient boosted decision trees. See Appendix~\ref{asec:emp:datasets} for details on data.
Note that this only ensures room for improvement for HGB, while highlighting cases with no room for improvement for SVM and MLP as we see in our results.
\paragraph{(Dis-)Regarding other baselines.}
While there are some existing schemes for {\sf FL-HPO}\xspace (as discussed in \S \ref{sec:related}), we are unable to compare {\sf FLoRA}\xspace to them for the following reasons: (i)~As noted by \citet[Section 1, Related Work]{khodak2021federated}, existing schemes focus ``on a small number of
hyperparameters (e.g. the step-size and sometimes one or two more) in less general settings (studying small-scale problems or assuming server-side validation data)'' whereas we explicitly assume no access to such a ``server-side validation data''. (ii)~Furthermore, we noted in \S \ref{sec:prob-setup} (\cond{1}), we do not assume any ``weight-sharing'' type capability, and hence it is not clear how FedEx~\citep{khodak2021federated} can be applied to {\sf FL-HPO}\xspace in the general\footnote{Moreover, \citep{khodak2021federated} claim that FedEx can handle architectural hyper-paramters but it is never demonstrated and discussed explicitly. In contrast, our proposed algorithm can handle architectural hyperparameters (as we do with HGB (tree depth) and MLP (width of the layer)).}.
\paragraph{Implementation.}
We consider two implementations for our empirical evaluation. In our first three sets of experiments, we emulate the final FL (Algorithm~\ref{alg:fl-hpo-lsa}, line 8) with a centralized training using the pooled data. We chose this implementation because we want to evaluate the final performance of any HP configuration (baseline or recommended by {\sf FLoRA}\xspace) in a statistically robust manner with multiple train/validation splits (for example, via 10-fold cross-validation) instead of evaluating the performance on a single train/validation. This form of evaluation is extremely expensive to perform in a real FL system and generally not feasible, but
allows us to evaluate how the performance of our single-shot HP recommendation fairs against that of the best-possible HP found via a full-scale centralized HPO.
\paragraph{Evaluation metric.}
In all datasets, we consider the balanced accuracy as the metric we wish to maximize. For the local per-party HPOs (as well as the centralized HPO we execute to compute the regret), we maximize the 10-fold cross-validated balanced accuracy. For Table~\ref{tab:baseline-comp}-\ref{tab:num-parties-comp}, we report the {\em relative regret}, computed as $\nicefrac{(a^\star - a)}{(a^\star - b)}$, where $a^\star$ is the best metric obtained via the centralized HPO, $b$ is the result of the baseline, and $a$ is the result of the HP recommended by {\sf FLoRA}\xspace. The baseline has a relative regret of 1 and smaller values imply better performance. A value larger than 1 implies that the recommended HP performs worse than the baseline.
\paragraph{Comparison to single-shot baseline.}
In our first set of experiments for 3-party {\sf FL-HPO}\xspace ($p = 3$), we compare our proposed scheme with the baseline across different datasets, machine learning models and {\sf FLoRA}\xspace loss surfaces. The aggregated results are presented in Table~\ref{tab:baseline-comp}, with the individual results detailed in Appendix~\ref{asec:baseline-comp}. For each of the three methods, we report the aggregate performance over all considered datasets in terms of (i)~inter-quartile range, (ii)~Wins/Ties/Losses of {\sf FLoRA}\xspace w.r.t. the single-shot baseline, and (iii)~a one-sided Wilcoxon Signed Ranked Test of statistical significance with the null hypothesis that the median of the difference between the single-shot baseline and {\sf FLoRA}\xspace is positive against the alternative that the difference is negative (implying {\sf FLoRA}\xspace improves over the baseline). Finally, we also report an ``Overall'' performance, further aggregated across all ML models.

All {\sf FLoRA}\xspace loss surfaces show strong performance w.r.t the single-shot baseline, with significantly more wins than losses, and 3rd-quartile relative regret values less than 1 (indicating improvement over the baseline). All {\sf FLoRA}\xspace loss surfaces have a p-value of less than $0.05$, indicating that we can reject the null hypothesis.
Overall, APLM shows the best performance over all loss surfaces, both in terms of Wins/Ties/Losses over the baseline as well as in terms of the Wilcoxon Signed Rank Test, with the highest statistic and a p-value close to $10^{-3}$. APLM also has significantly lower 3rd-quartile than all other loss surfaces. MPLM appears to have the worst performance but much of that is attributable to a couple of very hard cases with SVM (see Appendix~\ref{asec:baseline-comp} for detailed discussion). Otherwise, MPLM performs second best both for {\sf FL-HPO}\xspace with HGB and MLP.
\begin{table}[t]
\centering
\caption{Effect of increasing the number of parties on {\sf FLoRA}\xspace with different loss surfaces for HGB.
}
{\footnotesize
\begin{tabular}{lcccccc}
\toprule
Data                     & $p$      &$\gamma_p$& SGM & SGM+U & MPLM & APLM \\
\midrule
EEG Eye State            & 3        &  1.01 & 0.14   & 0.12   & 0.11   & 0.12 \\
14980 samples            & 6        &  1.01 & 0.07   & 0.00   & 0.07   & 0.09 \\
                         & 10       &  1.03 & 0.08   & 0.00   & 0.16   & 0.01 \\
                         & 25       &  1.08 & 0.35   & 0.92   & 0.17   & 0.04 \\
                         & 50       &  1.20 & 0.20   & 0.23   & 0.67   & 0.12 \\

\midrule
Electricity               & 3       &  1.01 & 0.17   & 0.14   & 0.09   & 0.12 \\
45312 samples             & 6       &  1.01 & 0.25   & 0.21   & 0.18   & 0.13 \\
                          & 10      &  1.02 & 0.03   & 0.06   & 0.32   & 0.14 \\
                          & 25      &  1.04 & 0.40   & 0.42   & 1.42   & 0.89 \\
                          & 50      &  1.07 & 1.57   & 1.57   & 0.89   & 1.13 \\
                          & 100     &  1.14 & 1.45   & 1.47   & 0.48   & 1.11 \\
\midrule
Pollen                    & 3       &  1.02 & 0.43   & 0.54   &  0.43  & 0.69 \\
3848 samples              & 6       &  1.10 & 1.02   & 0.91   &  0.54  & 0.56 \\
                          & 10      &  1.16 & 1.05   & 0.73   &  0.75  & 1.12 \\
\bottomrule
\end{tabular}
}
\label{tab:num-parties-comp}
\end{table}
\paragraph{Effect of increasing number of parties.}
In the second set of experiments, we study the effect of increasing the number of parties in the {\sf FL-HPO}\xspace problem on 3 datasets and HGB. For each data set, we increase the number of parties $p$ up until each party has at least 100 training samples. We present the relative regrets in Table~\ref{tab:num-parties-comp}. It also displays
$\gamma_p := \nicefrac{\left(1 - \min_{i \in [p]} \mathcal{L}_\star^{(i)} \right) }{ \left(1 - \max_{i \in [p]} \mathcal{L}_\star^{(i)} \right)}$, where $\mathcal{L}_\star^{(i)} = \min_{t \in [T]} \mathcal{L}_t^{(i)}$ is the minimum loss observed during the local asynchronous HPO at party $i$. This ratio $\gamma_p$ is always greater than 1, and highlights the difference in the observed performances across the parties. A ratio closer to 1 indicates that all the parties have relatively similar performances on their respective training data, while a ratio much higher than 1 indicating significant discrepancy between the per-party performances, implicitly indicating the difference in the per-party data distributions.

We notice that increasing the number of parties does not have a significant effect on $\gamma_p$ for the Electricity dataset until $p=100$, but significantly increases for the Pollen dataset earlier (making the problem harder). For the EEG eye state, the increase in $\gamma_p$ with increasing $p$ is moderate until $p=50$.
The results indicate that, with low or moderate increase in $\gamma_p$ (EEG eye state, Electricity for moderate $p$), the proposed scheme is able to achieve low relative regret -- the increase in the number of parties does not directly imply degradation in performance. However, with significant increase in $\gamma_p$ (Pollen, Electricity with $p=50, 100$ and EEG Eye State with $p=50$), we see a significant increase in the relative regret (eventually going over 1 in a few cases).
In this challenging case, MPLM (the most pessimistic loss function) has the most graceful degradation in relative regret compared to the remaining loss surfaces.
\begin{figure}
\centering
\begin{subfigure}{0.45\textwidth}
\includegraphics[width=\textwidth]{images/T-fig.png}
\caption{\# local HPO rounds.}
\label{fig:aplm-T-dep}
\end{subfigure}
~
\begin{subfigure}{0.45\textwidth}
\includegraphics[width=\textwidth]{images/T-comm-fig.png}
\caption{\# (HP, loss) pairs communicated to aggregator}
\label{fig:aplm-comm}
\end{subfigure}
\caption{Effect of different choices on {\sf FLoRA}\xspace with the APLM loss surface for different methods and datasets. More results and other loss surfaces are presented in Appendix~\ref{asec:emp:Tdep} and \ref{asec:emp:Tcommdep}.}
\label{fig:aplm-ablation}
\end{figure}
\paragraph{Effect of different choices in {\sf FLoRA}\xspace.}
In this set of experiments, we consider {\sf FLoRA}\xspace with the APLM loss surface, and ablate the effect of different choices in {\sf FLoRA}\xspace on 2 datasets each for SVM and MLP. First, we study the impact of the thoroughness of the per-party local HPOs, quantified by the number of HPO rounds $T$ in Figure~\ref{fig:aplm-T-dep}. The results indicate that for really small $T$ ($<20$) the relative regret of {\sf FLoRA}\xspace can be very high. However, after that point, the relative regret converges to its best possible value. We present the results for other loss surfaces in Appendix~\ref{asec:emp:Tdep}.

We also study the effect of the communication overhead of {\sf FLoRA}\xspace for fixed level of local HPO thoroughness. We assume that each party performs $T=100$ rounds of local asynchronous HPO. However, instead of sending all $T$ (HP, loss) pairs, we consider sending $T' < T$ of the ``best'' (HP, loss) pairs -- that is, (HP, loss) pairs with the $T'$ lowest losses. Changing the value of $T'$ trades off the communication overhead of the {\sf FLoRA}\xspace step where the aggregators collect the per-party loss pairs (Algorithm~\ref{alg:fl-hpo-lsa}, line 5). The results for this study are presented in Figure~\ref{fig:aplm-comm}, and indicate that, for really small $T'$, the relative regret can be really high. However, for a moderately high value of $T' < T$, {\sf FLoRA}\xspace converges to its best possible performance. Results on other loss surfaces and further discussion can be found in Appendix~\ref{asec:emp:Tcommdep}.
\begin{table*}[t]
\centering
\caption{Performance of {\sf FLoRA}\xspace with the IBM-FL system in terms of the {\em balanced accuracy} on a holdout test set (higher is better). The baseline is still the default HP configuration of {\tt HistGradientBoostingClassifier} in {\tt scikit-learn}.}
{\footnotesize
\begin{tabular}{lccccccc}
\toprule
Data          & \# parties & \# training data per party & Baseline & SGM & SGM+U & MPLM & APLM  \\
\midrule
Oil spill       &  $3$   &$200$   & 0.5895 & \textbf{0.7374} & 0.5909 & 0.7061 & 0.7332 \\
\midrule
EEG eye state &  $3$ & $3,000$  & 0.8864 & 0.9153 & 0.9211 & \textbf{0.9251} & 0.9245 \\

\midrule
Electricity   &  $6$  & $4,000$     & 0.8448 & 0.8562 & \textbf{0.8627} & 0.8621 & 0.8624 \\

\bottomrule
\end{tabular}
}
\label{tab:fl-summary}
\end{table*}
\paragraph{Federated Learning testbed evaluation.}
We now conduct experiments for histrogram boosted tree model in a FL testbed, utilizing IBM FL library~\citep{ludwig2020ibm,ong2020adaptive},
More specifically, we reserved $40\%$ of oil spill and electricity and $20\%$ of EEG eye state as global hold-out set only for evaluating the final FL model performance. Each party randomly sampled from the rest of the original dataset to obtain their own training dataset.
We use the same HP search space as in Appendix~\ref{asec:emp:search-space}.
We report the balanced accuracy of any HP (baseline or recommended by {\sf FLoRA}\xspace) on a single train/test split. Given balanced accuracy as the evaluation metric, we utilize (1 - balanced accuracy) as the loss $\mathcal{L}_t^{(i)}$ in Algorithm~\ref{alg:fl-hpo-lsa}
Each party will run HPO to generate $T = 500$ (HP, loss) pairs and use those pairs to generate loss surface either collaboratively or by their own according to different aggregation procedures described in \S \ref{sec:method:lsa}.
Once the loss surface is generated, the aggregator uses Hyperopt~\citep{bergstra2011algorithms} to select the best HP candidate and train a federated XGBoost model via the IBM FL library using the selected HPs.
Table~\ref{tab:fl-summary}
summarizes the experimental results for $3$ datasets, indicating that {\sf FLoRA}\xspace can significantly improve over the baseline in IBM FL testbed.

\section{Conclusions} \label{sec:next-steps}
\begin{comment}
{\tiny
\begin{itemize}
\item More datasets, methods
\item Extend single-shot to $K$-shot
\item Privacy
\item Improved performance via coordination between the local HPOs in each party
\item Handle distinction between global and local HPs for model (potential personalization)
\item Handle aggregator HPs
\end{itemize}
}
\end{comment}
How to effectively select hyper-parameters in FL settings is a challenging problem. In this paper, we introduced {\sf FLoRA}\xspace, a single-shot {\sf FL-HPO}\xspace algorithm that can be applied to a variety of ML models. We provided a theoretical analysis which includes a bound on the optimality gap incurred by the hyper-parameter selection performed by {\sf FLoRA}\xspace.  Our experimental evaluation shows that {\sf FLoRA}\xspace can effectively produce hyper-paramater configurations that outperform the baseline with just a single shot.

redpajama_set_name: RedPajamaArXiv


When I was asked to keynote a CSO event four years ago, I was pleasantly surprised to find that security culture was the top concern of the CSOs in attendance. Having performed many security assessments and penetration tests, I can tell you it is sadly obvious that even the best technical security efforts will fail if the company has a weak security culture. At the time, I was heartened to see that CSOs were moving past straight technological solutions and towards instilling a strong security culture as well.
In the intervening years, the perceived importance of security awareness programs has seemed to grow exponentially. And the resources allocated to them have increased as well.
Here is what I would consider the most relevant elements to integrate into a security awareness program.
Having C-level support inevitably leads to more freedom, larger budgets and increased support from other departments. Anyone responsible for running a security awareness program should first at least attempt to obtain strong support before focusing on anything else.
Yes, getting this level of support can be difficult, but there are certain best practices that will improve your chances of success, including highlighting the fact that security awareness is required for compliance and that awareness efforts will inevitably save the company money. Creating materials specifically for executives, such as newsletters and short articles highlighting relevant news and tips can also help garner that much-needed support.
Successful awareness programs find a way to involve other departments, such as legal, compliance, human resources, marketing, privacy and physical security. While it is easier to get this support if you already have C-level support, these departments frequently have mutual interests and might be amenable to providing additional resources, such as funding or distribution. Frequently, these departments can make security awareness efforts mandatory. For example, the legal and compliance departments carry a great deal of influence throughout the organization and can make security awareness a required component of other processes, such as new hire indoctrination.
To obtain this support, you might have to incorporate the needs of the cooperating departments with the general security awareness efforts. For example, you might suggest that you can use a security awareness newsletter to include compliance content. If it gets you the support you need, the effort is definitely worth the trouble.
It is also worth noting that most organizations require the involvement of other departments. For example, you may need to have corporate communications approve and distribute materials to employees; they likely have policies that govern how materials are to be distributed and the formats of those materials. You need to discover issues like this as quickly as possible.
It seems like most awareness programs are a standard check-the-box program, and content is driven by a list of potential computer-based training videos. As was demonstrated by the attempted Syrian Electronic Army attack against this publication, awareness programs that focus on timely information can be successful and prevent attacks.
The attacks don't have to be imminent against your own organization, though. There is plenty of fodder for relevant information. WannaCry was an excellent example of a cybersecurity related issue that received mainstream attention. Hacks against major retailers are another example of security issues made mainstream. Your awareness program should make regular use of these attacks to demonstrate the relevance of your efforts. This, in turn, motivates your users to follow your advice.
One of the key factors in having a successful effort is being able to prove that your effort is successful. The only way to do this is to collect metrics prior to initiating new awareness efforts. Without establishing a baseline, it is hard to demonstrate that your efforts had more than assumed success.
The metrics can include surveys on attitudes. They could also include the use of phishing simulation tools before and after awareness training. You can also examine the number of security related incidents reported to the help desk, the number of virus incidents, or reports from a webcontent filter that give the number of attempted visits to banned websites. When you can show measurable improvements in any aspect of security, you can better justify your program and obtain additional funding and support. Just about every department in a company has to prove its value, and security should not expect to be an exception.
Too frequently security departments seem to be the "Department of No". They focus on telling people what they shouldn't do, when the reality is that people will find a way to do whatever they want to do. While I acknowledge that there are clearly some actions that should not be allowed, those should be the exception rather than the rule.
Awareness efforts that focus on how to accomplish actions safely are more successful than those that focus on telling people not to do things. Ideally, awareness programs should tell people how to interact with information securely both at the office and at home. For example, instead of telling employees that they shouldn't be on social networks, teach them how to safely use social networks.
Create a reward structure that provides incentives based on actual behaviors demonstrated by employees. While it might not be practical for all organizations to implement this type of comprehensive gamification program, there is still the opportunity to implement some incentives and reward people for appropriate security behaviors.
While there is a place for computer-based training modules, too many programs rely on them completely as an awareness program. The most successful programs incorporate a variety of awareness tools, including newsletters, posters, games, newsfeeds, blogs, phishing simulations, etc. The most participative efforts appear to have the most success.
Another issue to consider is that materials should take into account the different demographics of your users. Diversify your materials to appeal to as many users as possible. There is definitely no such thing as "one size fits all" security awareness.
This is by no means an exhaustive list, but an excellent starting point. Remember, habits drive security culture, and there are no technologies that will ever make up for poor security culture. Awareness programs, when properly executed, provide knowledge that instills behavior. While most security professionals believe that good security behavior is a matter of common sense, the reality is that common sense is based upon common knowledge that needs to be made even more common.


text:
Baronia és una regió històrica de Sardenya nord-oriental, dins la província de Nuoro, que limita amb les subregions sardes de Gal·lura, Montacuto i Barbagia di Nuoro. Comprèn els municipis de Galtellì, Irgoli, Loculi, Lodè, Onifai, Orosei, Posada, Budoni, Siniscola i Torpè.

Durant l'edat mitjana constituí la part meridional del Jutjat de Gallura, i quan passà a la Corona d'Aragó fou dividida en la Baronia di Posada i la Baronia di Galtellì e Orosei.

Geografia de Sardenya

redpajama_set_name: RedPajamaWikipedia


text:
The EU Is Creating A Hostile Work Environment... For Employers
Capital Flows Contributor
Guest commentary curated by Forbes Opinion. Avik Roy, Opinion Editor.
Guest post written by
Pete Sepp
Mr. Sepp is President of National Taxpayers Union (ntu.org).
Labor lawyers have long been familiar with the term "hostile work environment"--conditions that make it impossible for an employee to do their job. Now tax lawyers at American businesses are contending with their own large-scale version of this problem, due to the policies of governments.
A case in point is the European Union's (EU) arbitrary crackdown on American multinationals operating there. Lately the European Parliament's tax committee as well as the European Commission have been pursuing a "reform agenda" to investigate (and likely overturn) previous, mutually agreed-upon tax payment structures between multinational companies and member countries. The majority of the targets are household names in the U.S., among them Amazon, Apple, Coca-Cola, Disney, Facebook, Google, McDonald's, Microsoft and Starbucks.
European Commission President Jean-Claude Juncker (L) and Economic and Financial Affairs commissioner Pierre Moscovici (R) attend a 2015 hearing organized by European Parliament's special committee on tax rulings. (JOHN THYS/AFP/Getty Images)
The hypocrisy is striking. On one hand, EU officials have been hounding American firms with regulatory threats to take away their toeholds in overseas markets, all in the name of enforcing what they call "competition." On the other hand, they seek to destroy competition among member nations' tax policies--not only with investigative maneuvers, but also by backing "harmonization" schemes that free-market organizations have called massive tax grabs in disguise.
Not to be outdone, tax collectors from individual EU states have behaved like IRS agents on steroids. This summer French and Spanish authorities raided Google's offices, seeking to justify cases for clawing back taxes from previous years (McDonald's got hit in France as well). Both nations have also proposed creating "virtual" tax bills for companies that may not have an actual physical presence within their borders--a serious breach of the basic protections that taxpayer advocates in this country have been defending. At the heart of this drama is crass politics… and, of course, money. EU taxation commissioner Pierre Moscovici recently told the media that lower taxes amount to funds "taken from … vital public services," a refrain often heard from profligate politicians on this side of the Atlantic.
At least some of our elected officials have caught on to the EU's shakedowns. In May, Senate Finance Committee Members led by Orrin Hatch (R-UT) and Ron Wyden (D-OR) urged U.S. Treasury Secretary Jack Lew to "protect U.S. interests" against EU harassment (four of the five company-specific EU investigations were aimed at America). As the Senators' letter to Treasury Secretary Lew rightly pointed out, any "retroactive application of a 'reform agenda' is improper and plainly undermines legal certainty and the rule of law."
Surely these Senators and their colleagues also recognize the root of that problem is embedded here: The firms are, after all, employing these strategies mainly to cope with punitive American taxes. At 39.1%, the U.S. has the highest statutory corporate tax rate in the industrialized world, and one of the worst effective rates after deductions and credits are taken into account. Recently, the Treasury issued a sweeping rule against "inversions" of U.S. companies relocating their headquarters to shift some income into less harshly-taxed countries. On its own, that edict could heap tens or hundreds of millions in compliance and business restructuring burdens on each affected company.
The bottom line is increasingly alarming for our job creators, which should concern all of us. Tens of millions of mutual fund holders depend on multinationals to balance their portfolio returns, while tens of thousands of small businesses participate in the multinationals' supplier networks. According to research by Matthew J. Slaughter for the Business Roundtable and U.S. Council for International Business, more than 28 million Americans were employed at multinationals, while just over one-fourth of these firms had U.S. parents classified as small or medium establishments by the federal government.
Gallery: 2016 30 Under 30 Europe: Finance
Many Americans aren't sympathetic to this plight. But if we won't shed any tears for multinationals, neither should we harbor any illusions; unless we think smarter about the tax trap in which our businesses are becoming ensnared, we will all be suffering the consequences.
Getting out of that trap begins with a total overhaul of the U.S. business tax system, one that emphasizes a lower, competitive rate, a simpler income base, protections against double taxation of earnings abroad, and transparent enforcement. Like any hostile work environment, today's hostile tax environment is something we need not tolerate, and must change.

redpajama_set_name: RedPajamaCommonCrawl


If you are looking for a diet that maintains a healthy lifestyle and yet has the rich flavor of some comfort foods you should try souping. This recent article published by CNN and written by Lisa Drayer highlights how you can make the most of your diet by consuming soup.
To read the article click here and then come on back for some top quality, premium soups!!


text:
Hypnotist
By Matt Kettmann
Wed Aug 09, 2006 | 11:45am
Joshua Seth at Centerstage
When most people think about onstage
hypnosis, embarrassing images of grown-ups frolicking like animals
come to mind. But according to internationally acclaimed stage
hypnotist Joshua Seth, who's putting on a show at the Center Stage
this Sunday, "You won't see anybody quacking like a chicken or
barking like a dog. What you are going to see is unbound creativity
let loose from the subconscious minds of the people I
hypnotize."
A registered hypnotherapist whose psychologist parents opened
his mind early on, Seth uses his "rapid, modern hypnosis technique"
to free the "inner creativity" of those audience members who
volunteer in order to "live their dreams." Sound goofy? Not so much
when Seth explains that he hypnotizes people into believing that
they've won the lottery or are driving their dream car. "You see
people really going through that unrestricted emotional release,"
Seth explained. In so doing, the volunteers and audience members
are taught that what seems unattainable might actually be
achievable. So powerful and entertaining is Seth's show that he was
once given a primetime slot on Japanese television, where he'd
hypnotize people on the street, including a young, dentist-phobic
girl whom he convinced to get two abscessed teeth yanked without
painkillers. An American version is currently in the works, said
Seth, who just moved to Santa Barbara and is doing this show to
build hype for a possible performance at the Lobero or the
Arlington.
Still not sure? Then take Seth's promise to heart: "Anybody that
comes in doubting leaves a believer." He takes Center Stage on
August 13, at 7:30 p.m. Call 963-0408, see centerstagetheater.org, or check out joshuaseth.com for more
https://www.independent.com/2006/08/09/hypnotist/
Matt Kettmann


text:
Moscow reacts to US sanctions
Russia's foreign ministry has vowed to respond in a "strong, not necessarily symmetrical" manner
FILE PHOTO. © AFP / JOEL SAGET
Russia's foreign ministry warned in a statement on Wednesday that there will be "a strong response" to new US sanctions. President Joe Biden announced on Tuesday that Washington would impose punitive measures on Russia for its formal recognition of two breakaway Ukrainian regions as independent nations.
"There shouldn't be any doubt – there will be a strong response, not necessarily symmetrical, but a calculated one whose impact the US will feel," Russia's Ministry of Foreign Affairs warned.
Russia's response points out that Moscow sees Washington's latest move as a "continuation of America's attempts to change Russia's course." It also notes that the US persists with sanctions "despite the obvious futility of the efforts undertaken for many years" in its bid to hinder Russia's economic development.
READ MORE: UN secretary-general under Western pressure on Ukraine – Russia
The ministry pointed out that Russia has proven that it can "minimize the impact" of punitive measures. If the US expects its sanctions to "affect Russia's determination to defend its interests," it is in for a disappointment, as that will never happen, it added.
On Tuesday, US President Joe Biden announced the imposition of economic sanctions on some of Russia's largest financial institutions along with "Russia's elite and [their] family members." Washington also warned Moscow that it would face yet more punitive measures should it "go further" with what Biden claimed was an "invasion" of Ukraine.
Biden's announcement came off the back of Russian President Vladimir Putin's decision late on Monday to recognize the Donbass' two breakaway republics as independent nations. On Tuesday, the Russian parliament granted Putin the authority to deploy Russian military personnel overseas; however, no decision regarding a peacekeeping mission in the Donbass has been announced yet.
READ MORE: Biden targets 'Russia's elite'
The crisis in Ukraine began in 2014, following the events of the Maidan, when violent street protests toppled the democratically elected government. This led to a conflict in the east of the country, as two regions broke from Kiev's control and declared themselves to be the Donetsk (DPR) and Lugansk (LPR) People's Republics.
Anti-Russian sanctionsDonbassRussia-US relationsUkraine


Grandmaster Cockroach have just released their second single "Oh! Whimsy" from their upcoming album "Music for Lousy Speakers" which is going to be out soon.
Lyrics are available on the bottom of this post. For more info read their press release below and if you want to check their first single "Ecstacy" also available through Downtuned Magazine click here.
Grandmaster Cockroach, release "Oh! Whimshy", the second single from their highly anticipated new album "Music for Lousy Speakers".
The lyrics were written first with political content but then turned to more inner spiritual themes, finally found its path to the ''music cosmos/universe''.
Dark, groovy and mellow with more emphasis on the vocals but still aggressive like the debut single "Ecstacy", 'Oh! Whimshy', which only boosts the anticipation for Grandmaster Cockroach's debut killer album!
Grandmaster Cockroach were formed in 2017 in Athens, with the idea that two people are enough to produce an original and at the same time complete sound.
Is something more to come?
Or something made from dawn?
'Cause there are more to come.
If there is beauty I'd like to go to.
But would you seize your guns.
If you don't want them why you control them?
When there are more to come.
I do know one thing. That I am insane with everything I love.
If you don't want them why you control them.
when there are more to come?
So don't act like a god.
If you don't want them, why do you control them?
If you don't want them why do you control them?
I do know one thing that I am insane with everything I love.
That I am in love with everything you're not.


Throughout the recent culture and science "wars," the radically new conceptions of knowledge and science emerging from such fields as the history and sociology of science have been denounced by various journalists, scientists, and academics as irresponsible attacks on science, absurd denials of objective reality, or a cynical abandonment of truth itself. In Scandalous Knowledge, Barbara Herrnstein Smith explores and illuminates the intellectual contexts of these crude denunciations. A preeminent scholar, theorist, and analyst of intellectual history, Smith begins by looking closely at the epistemological developments at issue. She presents a clear, historically informed, and philosophically sophisticated overview of important twentieth-century critiques of traditional—rationalist, realist, positivist—accounts of human knowledge and scientific truth, and discusses in detail the alternative accounts produced by Ludwik Fleck, Thomas Kuhn, Michel Foucault, Bruno Latour, and others.
With keen wit, Smith demonstrates that the familiar charges involved in these scandals—including the recurrent invocation of "postmodern relativism"—protect intellectual orthodoxy by falsely associating important intellectual developments with logically absurd and morally or politically disabling positions. She goes on to offer bold, original, and insightful perspectives on the currently strained relations between the natural sciences and the humanities; on the grandiose but dubious claims of evolutionary psychology to explain human behavior, cognition, and culture; and on contemporary controversies over the psychology, biology, and ethics of animal-human relations. Scandalous Knowledge is a provocative and compelling intervention into controversies that continue to roil through journalism, pulpits, laboratories, and classrooms throughout the United States and Europe.
Barbara Herrnstein Smith is Braxton Craven Professor of Comparative Literature and English at Duke University and Director of its Center for Interdisciplinary Studies in Science and Cultural Theory, and Distinguished Professor of English at Brown University. Among her books are Belief and Resistance: Dynamics of Contemporary Intellectual Controversy and Contingencies of Value: Alternative Perspectives for Critical Theory. Smith is a fellow of the American Academy of Arts and Sciences and the American Association for the Advancement of Science.