In addition, there is an excellent local labor force and the presence of over 300 freight forwarders in the airport vicinity makes freight operations a natural fit for this airport.The Airport Business section of our web site provides all of the information you need to conduct business at Orlando International Airport.
The Orlando-Sanford airport is northeast of downtown Orlando and Disney World and has 4 runways.
Shuttle transportation, multiple parking options and extremely competitive daily parking rates are key benefits of off airport parking for Orlando Sanford International Airport and any major airport and is a major reason why more than 3 million parking reservations have been made on this site instead of at the airport lots.
Orlando International Airport (IATA: MCO, ICAO: KMCO, FAA LID: MCO) is a major public airport located six miles (10 km) southeast of Downtown Orlando, Florida, United States. In compliance with Florida Statutes 316.1964, the Orlando Sanford International Airport provides courtesy parking within all of its pay lots to persons operating vehicles that display the Florida Toll Exemption Permit or that have been specially modified with equipment such as hand controls, lifts and ramps for utilization by a person who has a.
Hyatt Regency Orlando International Airport Located in the Main Terminal at Orlando International Airport (407) 825-1234.The Orlando-Sanford Airport is a medium sized, friendly, airport located approximately 30 miles northeast of the Orlando International Airport near the town of Sanford, Florida.View airlines serving the airport along with web site and contact information, baggage rules, and cities served.Parking at Plattsburgh International Airport is both inexpensive and convenient. Complete aeronautical information about Orlando Sanford International Airport (Orlando, FL, USA), including location, runways, taxiways, navaids, radio frequencies.
Airport parking at Orlando Sanford International Airport (SFB) Find on-site, off-site, short-term, long-term airport parking.
With over thirty-four million travelers each year, Orlando International Airport (MCO) is the thirteenth busiest airport in the US.Enjoy your trip with Sanford airport with economical packages.
INRIX receives parking information, including pricing, from many sources.The airport began during the 1940s as the Sanford Naval Air Station, and became the present civilian airport in 1968.Unlike Orlando International Airport where there are unlimited choices of off-site parking facilities available at below competitive rates, one can only assume, the parking authority at Sanford enjoys the revenue collected from their onsite lots, and apparently are not prepared to pass savings along to consumers by allowing off site.
Welcome to Maine and the Bangor International Airport (BGR) your gateway to all the best Maine has to offer. It is people like you that make folks like me want to stay with a company for the personal services.We also offer daily parker discounts for all Military, Police and Fire Personnel.
SFB Airport is situated about 24 miles to the northeast of downtown Orlando.
Budget Inn Sanford features RV and truck parking, self parking, and a microwave in the lobby.SFB is officially named Orlando Sanford International Airport, and is located in the city of Orlando.
IMPORTANT NOTICE: EXCLUSION OF LIABILITY SFB parking ticket is a license to use one (1) vehicle parking space at a specific revenue parking lot at the Orlando Sanford International Airport.Budget Inn Sanford International Airport This rating is a reflection of how the property compares to the industry standard when it comes to price, facilities and services available.There are 9 ways to get from Orlando Airport (MCO) to Orlando Sanford Airport (SFB) by train, bus, taxi, car, towncar or shuttle.Hotels Near Orlando Sanford International Airport (SFB) - Orlando Sanford International Airport (SFB) Accommodations close to Orlando Sanford International Airport (SFB) are provided in the list below ordered by distance (closest at the top).Orlando International Airport has an impressive range of facilities for the convenience of travellers, not least of which is the on-site Hyatt Regency hotel, which is located on the eastern side of the atrium in the main terminal building.Orlando Sanford International Airport (SFB) Hotel Guide Welcome to Orlando.
Parking at Southend Airport could not be easier, and made all the better by pre booking with Trusted Travel.
Orlando Sanford International Airport: Orlando Sanford International Airport is a public commercial airport located in Florida, near Orlando.We anticipate using our Overflow parking at times over the holiday weekend. please make note of our various parking locations.


text:
Eve Interviews
OCTOBER HOROSCOPES
Love Guru
About eve.com.mt
Writes Ramona Portelli on July 29, 2015
INTERVIEW WITH SARAH CAMILLERI
Date of Birth: 21st July, 1978
Where do you live? San Gwann
Status: Married
Star Sign: Cancer
Main occupation: Drama teacher
Media-related occupation: Actress/dancer/choreographer
SARAH CAMILLERI started dancing at the age of four. She gained her training under the supervision of Alison White and obtained a diploma in Modern Dance M.B.T.D.A. (British Theatre Dance Association), and in Classical Ballet M.B.T.D.A. She also obtained a Distinction in 'Gold Cup' in the Medal Test exams. From the age of fourteen, she was often asked to participate in various local television programmes, sometimes even as a resident dancer.
Sarah continued her studies at the University of Malta where she obtained a B.A. in Theatre Studies & Communications and obtained a P.G.C.E in Drama. She also studied musical theatre and acting under the supervision of Patrick Vella and obtained various certificates from Trinity College.
As a dancer, the list of performances in which Sarah has taken part is very long. To mention a few, her repertoire includes six pantomimes with MADC, the Malta Music Awards, the Malta Fashion Awards and Malta Song for Europe. She formed part of the Alison White Dancers and performed regularly in cabaret shows, which also involved performing on a cruise ship for over ten years. Over the years, she has choreographed and performed in seventeen musicals and a number of concerts
As an actress, Sarah has held both main and supporting roles on local TV dramas, including Is-Salib tal-Fidda, Nru 9, Dellijiet, Deċeduti, Iċ-Ċaqqufa, Dejjem Tiegħek Becky and Eklissi. She is currently playing the part of Emma in Vittmi, which is on every Sunday evening.
Sarah has also taken part in many local theatre productions, such as Il-Mizien Fetaħ Għajnejh, Ħafna Frattarija Għalxejn (Shakespeare's Much Ado About Nothing), Disney Re-tuned, Shadows and Gods, In Search of Simone, Bariona, and her latest and most favourite production, Kristu f'Strada Stretta. She has also taken part in a number of short films, adverts and two feature films which were shown in local cinemas.
Sarah has also had the opportunity to study, act and dance in England, Italy, Vienna and Berlin which were amazing experiences. Today, she works at St Albert College where she prepares students for their Trinity College exams.
Can you describe your state of mind when you are acting?
When acting, I focus totally on the role I am playing. My life and I fade into the background. I become totally engrossed in the situation the character is in and I do my best to portray the emotions and relationships the character has with herself and with the other characters in that particular scene. This differs drastically whether its theatre or TV. It's like being someone else with a little infusion of yourself.
What's wrong with the world today?
In my opinion, the problems in today's world all stem from greed and self-obsession. People steal, sell what's not theirs, fight and kill for power, breed animals irresponsibly and mistreat them. They exploit women and children and destroy our countryside. This is miles away from the simple, spiritual and harmonious life we should be sharing in our beautiful world! This hurts me a lot.
What, in your opinion, are the three most important attributes in a man?
First and foremost, he mustn't feel superior to the people around him. Secondly, he must be fun to be with and should be able to make you laugh. I also think that he should respect the person he's with and be loyal.
What role does technology play in your life?
I'm not a technical person at all, and although I cannot live my life without it, it frustrates me when I need to ask for help. I hate repetition and what's not abstract, and I've never read an instruction book in my life. I tend to experiment with technology which at times gets me into trouble. I suppose I should say that I have a love/hate relationship with technology!
What percentage of your life do you spend shopping?
Very little. I don't have much time for that, so I go in and out of shops quite quickly and always leave my shopping for the last minute. This differs when I'm quite free and in the mood. I never plan to go shopping, and the only person I enjoy shopping with is my husband and if he cannot come with me, I really don't mind going alone.
How do you normally relax when it all gets too much?
I never completely relax and that's a fault. But when I'm not working, I like to dine out with my son and husband or share a bottle of wine. I love watching films and I do that very often. Meeting my closest friends for a laugh is always great. Gozo is also a relaxing location for me and anywhere near the sea since I'm in love with it. Going abroad offers me mental relaxation!
Contact The Love Guru
Eve TV
Just Publications
Eve Malta

redpajama_set_name: RedPajamaCommonCrawl


text:
\section{Introduction}
\label{sec:intro}
\input{sections/1_intro}

\vspace{-0.3cm}
\section{Background \& Motivation}
\label{sec:background}
\input{sections/2_background}

\section{The FedorAS Framework}
\label{sec:fedoras}
\input{sections/3_methodology}

\section{Evaluation}
\label{sec:evaluation}
\input{sections/4_evaluation}

\vspace{-0.2cm}
\section{Related Work}
\vspace{-0.2cm}
\label{sec:related_work}
\input{sections/5_related_work}

\vspace{-0.4cm}
\section{Conclusion}
\vspace{-0.2cm}
\label{sec:conclusion}
\input{sections/7_conclusions}

{\small
\bibliographystyle{plain}

\subsection{SuperNet Training}
\label{sec:supernet_training}

\noindent\circled{0} \textbf{Search space \& models.}
First, we define the search space in terms of a range of operators options per layer in the network that are valid choices to form a network. This search space resides as a whole on the server and is only partially communicated to participating clients of a training round to keep communication overheads minimal.
Specific models (i.e. paths) consist of selections of one operator option per layer, sampled in a single-path-one-shot manner \cite{guo2020_spos} on-device per local iteration.

\noindent\circled{1} \textbf{Subspace sampling.}
It is obvious that communicating the whole space of operators along with the associated weights to each device becomes quickly intractable, especially bearing in mind that communication is usually a primary bottleneck in Federated Learning \cite{kairouz2019advances,challenges_fl2020msp}. To this direction, {\tt FedorAS}{} adopts a uniform parameter size budget, $B_{\Phi_{\text{comm}}}$, and samples\footnote{Non-parametric operations are always sent downstream and layers without non-parametric options are prioritised to guarantee a valid network.} the search space for operators until this limit is hit (Eq.~\ref{eq:comms_limit}).
Setting the limit to half the size of a typical network deployed for a task worked sufficiently well in our experiments and in fact accelerated convergence (Sec.~\ref{sec:ablation}).
\begin{equation}
    \label{eq:comms_limit}
    \sum_{l=0}^{L}\sum_{o\in\mathbb{O}_l} \mathbbm{1}_{\hat{\mathbb{O}}_l}{(o)} \Phi_{\text{comm}}(o) < B_{\Phi_{\text{comm}}}, \hat{\mathbb{O}}_l\subseteq\mathbb{O}_l
\end{equation}
where $L$ is the number of layer in the supernet, $\mathbb{O}_l$ the candidate and $\hat{\mathbb{O}}_l$ the selected operations in layer $l$, $\mathbbm{1}$ a unit vector of valid operations and $\Phi_\text{comm}$ a measure of DNN size (e.g. \#parameters).

In terms of sampling strategies, we experimented with uniform operator sampling, which we found to work sufficiently well and provide uniform coverage over the search space.
Different strategies that prioritise the sampling of certain operations is left as future work.
Last, it is worth noting that a different subspace (not necessarily mutex) is selected for each participating client in each round.

\noindent\circled{2} \textbf{Client-side sampling \& local training.}
Participating clients receive the subspace sampled on the server, $\{ \hat{\mathbb{O}}_l \}_{l\in[0;L]}$, from which they sample a single operator on every layer. This constitutes a path along the supernet ($p_L$) representing a single model. For every batch, clients sample paths that do not surpass the assigned training budget $B^{\text{Tier}}_{\Phi_{\text{train}}}$. Throughout this work, we consider $\Phi_{\text{train}}(\cdot)$ to be a cost function that counts the FLOPs of a given operator.
This FLOPs limit is defined per tier so that a network does not exceed the capabilities of the target device.
Path sampling is performed randomly across the valid (locally available) operations of each layer to ensure uniform coverage of the search space (Eq.~\ref{eq:sampled-path}). Alternative strategies, such as probabilistic sampling are left as future work.
\begin{equation}
\label{eq:sampled-path}
    p_L = \bigcup_{l\in[0;L]} o_l \sim \mathcal{U}\{\hat{\mathbb{O}}_l \} \;\;\text{s.t.} \;\; \sum_{l=0}^{L}\Phi_{\text{train}}(o_l) < B^{\text{Tier}}_{\Phi_{\text{train}}}
\end{equation}
After having sampled the path, a model is created and gets a forward and backward pass from the input sample at hand, in a single-path one-shot manner \cite{guo2020_spos}. The number of samples passing through each operator are kept and communicated back to the server, along with the updates, for properly weighting updated parameters upon aggregation, as we will see next.
Operator updates coming from single clients are dropped for privacy reasons.
Last but not least, both the batch size and the number of local epochs are hyperparameters that alter the amount of local exploration happening at each client. A utility-overhead balance needs to be met, however, to ensure the overhead is kept minimal.

\noindent\circled{3}
\textbf{Aggregation with OPA.}
An operator gets stochastically exposed to clients data. This stems from subspace sampling and client-side path sampling. As such, naively aggregating updates in an equi-weighed manner (Plain-Avg) or total samples residing a client (FedAvg \cite{mcmahan2017communication}) would not reflect the training experience during a round. For this reason, we propose \texttt{OPA}, OPerator Aggregation, an aggregation method that weighs updates based on the relative experience of an operator across all clients that have updated that operator.
Concretely, for each operation we track the number of examples updating its weights for each client. Then, upon aggregation, for searchable operation $o_l$, we normalise over all the updates of that specific operator across clients. Formally put:
\begin{equation}
    \omega_g^{(t+1)}(o_l) = \sum_{i\in K}^{|k|}\frac{|D_{(i,o_l,t)}|}{\sum_{\forall j \in [0,k-1]}|D_{(j,o_l,t)}|} \omega_i^{(t)}(o_l)
\end{equation}
where $\omega_g^{(t)}(\cdot)$ are global weights, $\omega_i^{(t)}(\cdot)$ are local weights of client $i$ at global step $t$ and $|D(i,o_l,t)|$ is the number of samples having backpropagated through an operator $o_l$ for client $i$ in round $t$.

\subsection{Model Search \& Validation}
\label{sec:model_validation_search}

After training the supernet, a search phase is implemented to discover the best trained architecture per device tier. Models are sampled with NSGA-II~\cite{nsga2} and evaluated on a global validation set.

NSGA-II is an evolutionary multi-objective genetic search algorithm that combines the elitist principle with diversity perservation (crowding distance) for candidate selection that are crossed via tournament selection for subsequent mutation generation.
Our rationale behind selecting NSGA-II algorithm for our search is the fact that it is multi-objective and allows us for efficient space exploration with the goal of optimising for model size in a specific tier and accuracy. Other search algorithms can be used in place of NSGA-II, based on the objective at hand. At the end of this stage, we have a model architecture per tier, already encompassing knowledge across devices of different tiers, which serves as initialisation for further training.

\subsection{Fine-tuning Phase}
\label{sec:finetuning_phase}

Subsequently, we move to train each of the previously produced models in a federated manner across all eligible clients. This means that architectures falling in a specific tier in terms of footprint can be trained across clients belonging to that tier and above.
Our hypothesis here is that compared to regular FL where one needs to trade off capacity of participation, {\tt FedorAS}{} allows for better generalisation due to knowledge sharing through its supernet structure.
Here we use conventional FedAvg and partial client participation per round (see Section~\ref{sec:background}), again with the goal of training a single network per tier, but without the nuisance of stragglers.

\section{Broader Impact}

Our system, {\tt FedorAS}{}, performs federated and resource-aware NAS in the cross-device setting.
Despite the benefits illustrated compared to centralised NAS and cross-silo FL solutions, running Neural Architecture Search is still a resource-demanding process, in terms of compute, memory and network bandwidth.
While {\tt FedorAS}'s target devices can be of significantly lower TDP (i.e. smartphones and IoT devices vs. server-grade GPUs) -- with consequences on the overall energy consumption of training~\cite{qiu2021first} -- they are resources not directly owned by the operator.
As such, special care needs to be taken with respect to how many device resources are leveraged at any point in, so as not hinder the usability of the device or invoke monetary costs to the user~\cite{ieee_iot_fl_incentives}.

\section{Limitations \& Future Work}

Despite the challenges address by {\tt FedorAS}, our prototype has certain limitations. First and foremost, we have opted to cluster devices (i.e. in tiers) based on their FLOPS. While it is perfectly normal to divide clusters based on other criteria (e.g. memory, latency, energy) or in a multi-objective manner, we have kept it simple.
Moreover, one can opt for biased sampling of i) clients participating in a round, ii) the subspace they get communicated and iii) the paths sampled from that subspace, we opted for uniform sampling for all of the above for simplicity, uniform coverage of the search space and fairness in participation. We leave the exploration of such strategies as future work.
Last but not least, we have considered privacy-enhancing techniques, such as Differential Privacy~\cite{dp_mcmahan} or Secure Aggregation~\cite{Bonawitz2017-secagg,Bell2020-secaggpoly}, as orthogonal to our scheme. Combining Federated NAS with such strategies can expose interesting trade-offs of exploration-exploitation-privacy budgets that could be explored in the future.

\section{Detailed Experimental Setup}

\subsection{Datasets}

\textbf{CIFAR-10/100.} The CIFAR-10 datasets contains 10k and 50k 32x32 RGB images in its test and training sets respectively comprising ten classes.
The goal is to classify these images correctly. Similarly, CIFAR-100 follows an identical partitioning but this time across 100 classes (\textit{fine} lebels) or 20 superclasses (\textit{coarse} labels).
Both CIFAR datasets have a uniform coverage across their classes.

\textbf{SpeechCommands.} The Speech Commands datasets~\cite{speechcommands} is comprised of 105,829, 16KHz 1-second long audio clips of a spoken word (e.g. "yes", "up", "stop") and the task is to classify these in a 12 or 35 classes setting. The datasets comes pre-partitioned into 35 classes and in order to obtain the 12-classes version, the standard approach~\cite{berg21_interspeech, 2018arXiv180808929C,Vygon_2021} is to keep 10 classes of interest (i.e. "yes", "no", "up", "down", "left", "right", "on", "off", "stop", "go"), place the remaining 25 under the "unknown" class and, introduce a new class "silence" where no spoken word appear is the audio clip.  In this work we consider SpeechCommandsV2, the most recent version of this dataset.
The dataset spans three disjoint set of speakers: 2112 form the training set, 256 for validation and, 250 for testing. In {\tt FedorAS}{}, the supernet training phase makes uses of the 2112 clients in the training partition only. The results are obtained by measuring the performance of the discovered models on the data of the 250 clients comprising the test set. Data is not uniformly distributed and some clients have more data than others. This dataset is processed by first extracting MFCC~\cite{mfcc} features from each audio clip~\cite{zhang2018hello,berg21_interspeech}. Across our experiments we extract 40 MFCC features from a MelSpectrogram where each audio signal is first sub-sampled down to 8KHz and then sampled using 40ms wide time windows strided 20ms appart. This results in 1-second audio clip being transformed into a 1$\times$40$\times$51 input that can be passed to a CNN.

\textbf{Shakespeare.} This dataset is built~\cite{leaf,mcmahan2017communication} from \textit{The Complete Works of Willilam Shakespeare} and partitioned in such a way data that from each role in the play is assigned to a client. This results in a total of 1,129 partitions where the average number of samples per device is 3.7K and the standard deviation 6.2K samples. This makes Shakespeare a relatively imbalanced dataset. The task is to correctly predict the next character in a dialog given the previously seen characters in the sentence. The vocabulary considered has 86 English characters as well as four special tokens: start and end of sentence, padding and out-of-vocabulary tokens.

\subsection{Search Spaces}
\label{sec:search_spaces}

\textbf{CIFAR-10 and CIFAR-100.} We use a ResNet-like search space similar to the one used in, for example, Cai et al.~\cite{proxylessnas}.
Specifically, our model is a feedforward model with a fixed (\textit{i.e.} non-searchable) stem layer followed by a sequence of 3 searchable blocks, each followed by a fixed reduction block.
A standard block consists of 4 searchable layers organized again in a simple feedforward manner.
Operations within a standard block preserve shape of their inputs, but are allowed to change dimensions of intermediate results.
On the other hand, the goal of reduction blocks is to reduce spatial resolution ($2\times$ in each dimension) and increase number of channels ($1.5\times$).
Reduction blocks are fixed throughout and consists of a depthwise separable convolution $3\times3$, with the first part performing spatial reduction and the second increasing the number of channels, and a standard $2\times2$ convolution applied to a residual link.
The sequence of blocks is finished with a global average pooling per-channel and a classification layer outputting either 10 or 20/100 logits, which is the only difference between the two models.
Each convolution operation is followed by a batch normalization (BN) and ReLU.

 In our experiments we considered the following candidate operations:
 \begin{itemize}
     \item standard convolution $1\times1$ with BN and ReLU
     \item depthwise separable convolution $3\times3$ with expansion ratio (controlling the number of intermediate channels) set to $\{0.5, 1, 2\}$, with each expansion ratio being an independent choice in our search space
     \item MobileNet-like block consisting of a convolution with kernel size $k$ and expansion ratio $e$, followed by a Squeeze-and-Excitation layer, followed by a $1\times1$ convolution reverting the channels expansion, we considered $\{ (k=1,e=2), (k=3,e=0.5), (k=3,e=1), (k=3,e=2) \}$
     \item identity operation
 \end{itemize}

The stem layer was set to a $3\times3$ convolution outputting 64 channels.

\textbf{Speech Commands.} The model follows the one used for the CIFAR datasets but is made more lightweight -- to roughly match what can be found in the literature -- by reducing stem channels to 16 and including only 1 (resp. 2) searchable operations in the first (resp. last) two blocks.
Additionally, reduction block only includes a single $1\times1$ convolution, that changes the number of channels, followed by a $3\times 3$ average pooling that reduces spatial dimensions.
We also include additional candidate operations:
\begin{itemize}
    \item standard convolution $3\times 3$
    \item depthwise separable convolution with kernel size 1 and expansion ratio 2
\end{itemize}
All other operations from the CIFAR model are also included.

\textbf{Shakespeare.} We base our design on the model used by Horvath et al.~\cite{fjord}.
Specifically, the model is a recurrent network that begins with a fixed embedding layer outputting 8-dimensional feature vector per each input character.
This is then followed by a searchable operation \#1 that increases dimensionality from 8 to 128; in parallel, we have a searchable skip connection \#1 operation whose output is added to the output of the operation \#1.
Later there is another searchable operation \#2 with its own skip connection \#2, both keeping the hidden dimension at 128.
Again, their outputs are added and passed to the final classification layer.

Candidate operations for each of the four searchable layers are mainly the same, with minor adjustments made to make sure a valid model is always constructed.
These include:
\begin{itemize}
    \item an LSTM layer~\cite{lstm},
    \item a GRU layer~\cite{gru},
    \item a LiGRU layer with \texttt{tanh} activation~\cite{ligru},
    \item a QuasiRNN layer~\cite{quasirnn},
    \item a simple Linear layer (no recurrent connection) followed by a \texttt{sigmoid} activation,
    \item a 1D convolution with kernel 5 spanning time dimension (looking at the current character and 4 previous ones), followed by a \texttt{sigmoid} activation (no normalisation),
    \item identity operation (only included in later operation, after feature dimension has been increased to 128),
    \item zero operation, outputting zeros of the correct shape (only included in skip connection layers).
\end{itemize}

For LiGRU and QuasiRNN we used implementations provided by the Speechbrain project~\cite{speechbrain}, after minor modifications.
For others, we used standard operations from PyTorch.
All operations were used as unidirectional.
We did not use any normalisation throughout the model.

\begin{figure}
    \centering
    \includegraphics[width=\linewidth]{images/FedorasModels.pdf}
    \captionsetup{font=small,labelfont=bf}
    \caption{Summary of models used in our experiments: \textit{left)} CIFAR-10 and CIFAR-100, \textit{middle)} Speech Commands, \textit{right)} Shakespeare. Blocks highlighted in blue are fixed, orange blocks represent searchable layers -- see section~\ref{sec:search_spaces} for details. c -- output channels, s -- stride, d -- output feature dimension, DSConv -- depthwise separable convolution. Whenever a layer has more than one input they are added. All convolutions are followed by BN and ReLU. For convolution and pooling operations, numbers in parentheses represent window sizes.}
    \label{fig:models}
\end{figure}

\subsection{Tiers: definition and client assignment}
With {\tt FedorAS}{} the discovery and training of architectures happens in a tier-aware fashion as a federated process. In this work we considered splitting each search space along the FLOPs dimensions (but other splits are possible, e.g: energy, peak-memory usage, etc -- or a combination of these). Figure~\ref{fig:searchspaces_megaplot} illustrates the span in terms of model parameters and FLOPs of each search space presented in \ref{sec:search_spaces} and the split along the FLOPs dimensions for each of them. These search spaces vary considerably in terms of size and span, which motivated us to use different number of tiers or client-to-tier assignment strategies. Table~\ref{tab:tiers_definition} shows the the FLOPs ranges considered for each tier, the number of models in those sub-spaces exposed to {\tt FedorAS}{} as well as the ratio of models in the entire search space that fall onto each tier.

\begin{table}[h]
    \centering
    \small
    \captionsetup{font=small,labelfont=bf}
    \caption{This table summarises how each search space is split into tiers. Parameters $\rho_L$ and $\rho_H$ are used to conveniently split the FLOPs axis for each dataset and present challenging scenarios for {\tt FedorAS}{}. The last column refers to the percentage of the total clients that are assigned to each tier.}
    \scalebox{0.85}{
    \begin{tabular}{lcccccc}
        \toprule
         Dataset     & [$\rho_L$, $\rho_H$] &  Tier  & FLOPs range &  \# Models & Portion(\%) & Clients (\%)\\ \midrule
        \multirow{4}{*}{CIFAR-10}    & \multirow{4}{*}{[0.0, 0.95]} &   T1     & [0, 120.9M]        & 239.4$\cdot10^{12}$        &  12.92\% & 25\% \\
                                     &                              &   T2     & (120.9M, 223.2M]        & 1089.4$\cdot10^{12}$        &  58.80\% & 25\% \\
                                     &                              &   T3     & (223.2M, 325.4M]        & 477.2$\cdot10^{12}$        &  25.75\% & 25\% \\
                                     &                              &   T4     & (325.4M, 716.0M]        & 46.9$\cdot10^{12}$        &  2.53\% & 25\% \\
        \midrule
        \multirow{4}{*}{CIFAR-100}    & \multirow{4}{*}{[0.0, 0.9]} & T1     & [0, 111.5M]        & 168.4$\cdot10^{12}$        &  9.09\% & 25\% \\
                                     &                              &   T2     & (111.5M, 204.3M]        & 975.4$\cdot10^{12}$        &  52.64\% & 25\% \\
                                     &                              &   T3     & (204.3M, 297.1M]        & 609.5$\cdot10^{12}$        &  32.89\% & 25\% \\
                                     &                              &   T4     & (297.1M, 716.0M]        & 99.7$\cdot10^{12}$        &  5.38\% & 25\% \\
        \midrule
        \multirow{4}{*}{CIFAR-100$^\dagger$}    & \multirow{4}{*}{[0.0, 0.9]} & T1     & [0, 111.5M]        & 168.4$\cdot10^{12}$        &  9.09\% & 50\% \\
                                     &                              &   T2     & (111.5M, 204.3M]        & 975.4$\cdot10^{12}$        &  52.64\% & 25\% \\
                                     &                              &   T3     & (204.3M, 297.1M]        & 609.5$\cdot10^{12}$        &  32.89\% & 12.5\% \\
                                     &                              &   T4     & (297.1M, 716.0M]        & 99.7$\cdot10^{12}$        &  5.38\% & 12.5\% \\
        \midrule
        \multirow{4}{*}{SpeechCommands} & \multirow{4}{*}{[0.3, 0.925]} & T1     & [0, 5.0M]        & 827.5$\cdot10^{3}$        &  46.71\% & 80\% \\
                                        &                              &   T2     & (5.0M, 7.5M]        & 617.4$\cdot10^{3}$        &  34.85\% & 1.25\% \\
                                        &                              &   T3     & (7.5M, 10.1M]        & 260.2$\cdot10^{3}$        &  14.69\% & 1.25\% \\
                                        &                              &   T4     & (10.1M, 20.0M]        & 66.4$\cdot10^{3}$        &  3.75\% & 17.5\% \\
        \midrule
        \multirow{5}{*}{Shakespeare} & \multirow{5}{*}{[0.1, 0.77]} & T1     & [0, 7.8M]        & 316        &  13.48\% & 20\% \\
                                     &                              &   T2     & (7.8M, 12.8M]        & 577        &  24.54\% & 20\% \\
                                     &                              &   T3     & (12.8M, 17.8M]        & 787        &  33.48\% & 20\% \\
                                     &                              &   T4     & (17.8M, 22.8M]        & 473        &  20.14\% & 20\% \\
                                     &                              &   T5     & (22.8M, 33.8M]        & 196        &  8.37\% & 20\% \\

        \bottomrule
        \multicolumn{7}{l}{$\dagger$ For the \textit{multitask} setup.}
    \end{tabular}
}
    \vspace{2mm}
    \label{tab:tiers_definition}
\end{table}

\begin{figure*}
    \centering
    \begin{subfigure}[b]{0.485\textwidth}
        \centering
        \includegraphics[width=\textwidth]{images/enumerate_decisions_cifar10.pdf}
        \caption[]%
        {{\small CIFAR-10 }}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.485\textwidth}
        \centering
        \includegraphics[width=\textwidth]{images/enumerate_decisions_cifar100.pdf}
        \caption[]%
        {{\small CIFAR-100 }}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{0.485\textwidth}
        \centering
        \includegraphics[width=\textwidth]{images/enumerate_decisions_speechcommands.pdf}
        \caption[]%
        {{\small SpeechCommands }}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.485\textwidth}
        \centering
        \includegraphics[width=\textwidth]{images/enumerate_decisions_shakespeare.pdf}
        \caption[]%
        {{\small Shakespeare }}
    \end{subfigure}
    \captionsetup{font=small,labelfont=bf}
    \caption[ meh ]
    {\small For each search space, we randomly sample 500 architectures and color code them based on the tier they belong to. Vertical dashed lines represent the boundaries between device tiers. For Shakespeare, the majority of the candidate operators in the searchspace include just linear layers or no-op layers, as a results the number of FLOPs grows almost linearly with the model of parameters.}
    \label{fig:searchspaces_megaplot}
\end{figure*}

\textbf{Identifying FLOPs ranges for each tier.} Regardless of the dataset, we follow a common approach to divide the FLOPs dimension that each dataset spans. The main aim is to finely control how many models fall onto the smallest/largest tier, given that these are considerably sparse regions of the entire search space. If we were to evenly split the FLOPs dimension, almost no architecture would fall onto the largest tier. To simplify the process of defining where each tier's boundaries lie, we follow these steps which require two hyperparameters: (1) we construct a long array of FLOPs from models sampled from the search space and this array is then sorted from lowest to highest; then, (2) we normalise this array of FLOPs and compute the cumulative sum of such normalised array; finally, (3) we identify the lower limit of the highest tier as maximum FLOPs value found in the first $\rho N, \rho \in [0,1]$, elements of the array where $N$ is the number of samples taken (we found 100K to work well). Essentially, we find the FLOPs for which the approximated PDF over the FLOPS of a given search space doesn't surpass $\rho$. Once that FLOPs value is identified, the FLOPs range is evenly split for the tiers below and the higher limit for the largest tier becomes the maximum FLOPs a model in the search space can have. This is the approach for CIFAR-10 and CIFAR-100 as illustrated in Figure~\ref{fig:searchspaces_megaplot} (a) and (b). For SpeechCommands and Shakespeare we introduce a similar approach to give a wider range to the smallest tier. We refer to $\rho_L$ and $\rho_H$ as the PDF ratios to identify the boundaries splitting Tier1\&2 and Tier3\&4 respectively. The concrete values for each dataset as well as the FLOPs values for each tier boundary are shown in Table~\ref{tab:tiers_definition}.

\textbf{Clients to tiers assignment.} For CIFAR-10/100 clients are uniformly assigned to a cluster of devices or tier. For these datasets we consider four tiers, so each ends up containing 25\% the clients resulting in 25 clients for CIFAR-10 and 125 clients for CIFAR-100. Similarly, for Shakespeare clients are also uniformly assigned to a tier, resulting in 143 clients per tier. For SpeechCommands, we designed a more challenging setup and divide the clients into tiers as follows: 80\% of clients are assigned to be tier-1 devices, 17.5\% are Tier-4 devices and the rest is evenly split into Tier-2 and Tier-3. This distribution better represents the the types of systems in the wild that perform keyword-spotting~\cite{zhang2018hello}, where the majority of the commercially deployed systems run these applications on low end CPUs or microcontrollers due to their \textit{always-on} nature. Across datasets, the client-to-tier assignments was done irrespective of the amount of data these contained or distribution over the labels.

\subsection{Hyperparameters}

Here we present the hyperparameters used across all datasets and tasks to generate the results presented in \ref{sec:evaluation}. In Table~\ref{tab:train_hp} we show the hyperparamters utilised for the first stage in the {\tt FedorAS}{} framework: federated training of the supernet. After this stage, the best model for each tier is extracted from the supernet and then, they get finetuned in a per-tier aware manner (i.e. clients in tier $T$ and above can finetune a model that belong to tier $T$). The hyperparameters of these two consecutive stages are shown in Table~\ref{tab:searchandfinetune_hp}.

Regarding the End2End results (i.e. models discovered by {\tt FedorAS}{} but trained from scratch -- discarding the weights given by the supernet) we present the hyperparameters in Table~\ref{tab:end2end}. These hyperparameters were the ones used to generate the results in Table ~\ref{tab:results} as well as Figure~\ref{fig:capacity_vs_participation}. In Table \ref{tab:comparison} we present a CIFAR-100 result that largely outperforms existing baselines. This was achieved by a model discovered by{\tt FedorAS}{} but trained in End2End setting following the setup as in \cite{reddi2020adaptive}: 10 clients per round for 4k rounds using batch 20, starting learning rate of 0.1 decaying to 0.01 following a \textit{cosine} scheduling.

\begin{table}[h]
    \centering
    \small
    \captionsetup{font=small,labelfont=bf}
    \caption{Hyperparameters used for the Federated supernet training stage in {\tt FedorAS}{}, as described in \ref{sec:supernet_training}. The learning rate is kept fixed during this stage. In all datasets, the aggregation strategy followed the proposed histogram-informed methodology OPA, first presented in \ref{sec:supernet_training}.}
    \scalebox{0.85}{
    \begin{tabular}{lcccccccc}
        \toprule
        \multirow{2}{*}{Dataset}     &  \# Federated & \# Clients & \# Local & Local & Batch & \multirow{2}{*}{LR} & \multirow{2}{*}{Momentum} & Gradient\\
                    &  Rounds       & per round  & Epochs & Optimizer & Size & & & Clipping \\ \midrule
        CIFAR-10    &   500     & 10        & 50        & SGD & 128 & 0.1 & 0.9 & N \\
        CIFAR-100    &   500/750     & 10        & 25        & SGD & 64 & 0.1 & 0.9 & N \\
        SpeechCommands    &   750     & 21        & 25        & SGD & 64 & 0.1 & 0.9 & N\\
        Shakespeare    &   500     & 16        & 5        & SGD & 4 & 1.0 & 0.0 & Y\\

        \bottomrule
    \end{tabular}
    }
    \vspace{2mm}
    \label{tab:train_hp}
\end{table}

\begin{table}[h]
    \centering
    \small
    \captionsetup{font=small,labelfont=bf}
    \caption{Hyperparameters used by {\tt FedorAS}{} to search for the best model in the supernet and finetune them in a tier-aware fashion as described in \ref{sec:model_validation_search} and \ref{sec:finetuning_phase} respectively. Searching iterations shown are allocated per-tier (i.e. CIFAR-10 has 4 tiers so a total of 4K valid models would be considered during the search). \textit{Cosine} LR scheduling gradually reduces the initial LR (shown in the table) by an order of magnitude over the span of the the finetuning process. For Shakespeare, \textit{step} LR decay worked best. This is applied at rounds 50 and 75, each decreasing the LR by a factor of 10$\times$. SpeechCommands assigns different search iterations based on the ratio of clients assigned to each tier. {\tt FedorAS}{} scales the amount of search for valid models within the tier accordingly. For Shakespeare, the sub-searchspaces that yield Tier-1 and Tier-5 models are smaller than for the other tiers, we therefore consider fewer search iterations.}
    \scalebox{0.825}{
    \begin{tabular}{lcccccccccc}
        \toprule
        \multirow{2}{*}{Dataset}     &  \# Tiers & \# Search & \# Finetune & \# Clients & Local & Batch & \multirow{2}{*}{LR} & LR & Other\\
                    &         & Iterations  & Rounds & per round & Epochs & Size & & Scheduling & Hyperparams \\ \midrule
        CIFAR-10    &   4     &  1000        & 100        & 6 & 1 & 32 & 0.01  & cosine & momentum=0.9\\
        CIFAR-100    &   4     & 1000       & 100        & 6 & 1 & 32 & 0.01  & cosine & momentum=0.9\\
        SpeechCommands    &   4     & 500/2000       & 100        & 21 & 1 & 32 & 0.01 & cosine & momentum=0.9\\
        Shakespeare    &   5     & 100/150        & 100        & 16 & 1 & 4 & 1.0 & step & g.clipping=5\\

        \bottomrule
    \end{tabular}
    }
    \vspace{2mm}
    \label{tab:searchandfinetune_hp}
\end{table}

 \begin{table}[h]
    \centering
    \small
    \captionsetup{font=small,labelfont=bf}
    \caption{The architectures discovered by {\tt FedorAS}{} can also be trained from scratch. This table contains the hyperparameters utilised to generate the End2End results shown in table~\ref{tab:comparison} and table~\ref{tab:results}. Training of these baselines is also peformed in a tier-aware fashion. \textit{Cosine} LR scheduling gradually reduces the initial LR (shown in the table) by an order of magnitude over the span of the the finetuning process. For Shakespeare, \textit{step} LR decay worked best. This is applied at rounds 250 and 375, each decreasing the LR by a factor of 10$\times$. }
    \scalebox{0.98}{
    \begin{tabular}{lcccccccccc}
        \toprule
        \multirow{2}{*}{Dataset}     &  \# Federated & \# Clients & Local & Batch & \multirow{2}{*}{LR} & LR & Other\\
                    &   Rounds & per round & Epochs & Size & & Scheduling & Hyperparams \\ \midrule
        CIFAR-10    & 500        & 10 & 1 & 32 & 0.1  & cosine & momentum=0.9\\
        CIFAR-100    & 500       & 10 & 1 & 32 & 0.1  & cosine & momentum=0.9\\
        SpeechCommands & 500     & 21 & 1 & 32 & 0.1 & cosine & momentum=0.9\\
        Shakespeare    & 500     & 16 & 1 & 4 & 1.0 & step & g.clipping=5\\

        \bottomrule
    \end{tabular}
    }
    \vspace{2mm}

    \label{tab:end2end}
\end{table}

\section{Comparison against random search}

In the main paper we compare our {\tt FedorAS}{} to the state-of-the-art models from the literature (Sec.~\ref{sec:fedoras_vs_sota}) and also investigate the impact of initializing models with weights from a supernet (Sec.~\ref{sec:supernet_vs_random_init}).
Here we present one more important baseline for any NAS algorithm - comparison to a random search.
Specifically, while we have already shown that models found by ${\tt FedorAS}{}$ in most cases benefit from supernet training, this does constitute a conclusive argument justifying the necessity of the entire process in a broader context.
Perhaps there exist models that do not need to be initialized from a supernet but, since the focus of our work is on supernet training, we missed them due to biased conditioning?
In order to check this hypothesis we run a simple random search algorithm, to get an estimate of the expected \emph{best case} performance of models from our search spaces when trained following standard FL procedure.
This simple baseline was allowed to train random models until the total training cost exceeded the cost of running {\tt FedorAS}{} -- this turned out to be equivalent to roughly 40 fully-trained models.
After we run out of the training budget, we simply get the best model for each tier as our solution to NAS.
We repeated the entire process 3 times for each search space and report average and standard deviation compared to the average performance of models found by {\tt FedorAS}{} in Tab.~\ref{tab:fedoras_vs_random}.
Additionally, we also plot detailed performance of each model trained during this process in Fig.~\ref{fig:equi_budget} for the sake of completeness.

\begin{table}
    \centering
    \small
    \captionsetup{font=small,labelfont=bf}
    \caption{Comparison of performance of models found with {\tt FedorAS}{} and with a simple random search.}
    \scalebox{0.68}{
    \begin{tabular}{lcccccccccc}
        \toprule
        \multirow{2}{*}{Dataset}     & \multicolumn{5}{c}{{\tt FedorAS}{}} & \multicolumn{5}{c}{Random search} \\ \cmidrule(lr){2-6} \cmidrule(l){7-11}
                                     & T1 & T2 & T3 & T4 & T5 & T1 & T2 & T3 & T4 & T5 \\
        \midrule
        CIFAR-10 {\scriptsize ($\alpha=1000$)} & 89.40{\scriptsize$\pm$0.19} & 89.60{\scriptsize$\pm$0.15} & 89.64{\scriptsize$\pm$0.22} & 89.24{\scriptsize$\pm$0.29} & -                          & 86.03{\scriptsize$\pm$0.13} & 86.39{\scriptsize$\pm$0.28} & 85.18{\scriptsize$\pm$0.26} & 79.84{\scriptsize$\pm$0.75} & - \\
        CIFAR-10 {\scriptsize ($\alpha=1.0$)}  & 85.99{\scriptsize$\pm$0.13} & 86.30{\scriptsize$\pm$0.41} & 86.34{\scriptsize$\pm$0.19} & 85.58{\scriptsize$\pm$0.55} & -                          & 83.68{\scriptsize$\pm$0.25} & 84.50{\scriptsize$\pm$0.31} & 84.30{\scriptsize$\pm$0.38} & 79.65{\scriptsize$\pm$0.18} & - \\
        CIFAR-10 {\scriptsize ($\alpha=0.1$)}  & 81.01{\scriptsize$\pm$0.46} & 81.53{\scriptsize$\pm$0.29} & 80.64{\scriptsize$\pm$0.66} & 80.85{\scriptsize$\pm$0.28} & -                          & 66.75{\scriptsize$\pm$0.74} & 70.85{\scriptsize$\pm$1.53} & 69.00{\scriptsize$\pm$0.18} & 66.08{\scriptsize$\pm$2.29} & - \\
        CIFAR-100                              & 45.25{\scriptsize$\pm$0.13} & 45.84{\scriptsize$\pm$0.18} & 45.42{\scriptsize$\pm$0.39} & 45.07{\scriptsize$\pm$0.71} & -                          & 34.41{\scriptsize$\pm$0.90} & 37.64{\scriptsize$\pm$0.46} & 36.98{\scriptsize$\pm$0.64} & 35.45{\scriptsize$\pm$0.90} & - \\
        SpeechCommands                         & 80.19{\scriptsize$\pm$1.78} & 80.47{\scriptsize$\pm$1.69} & 81.00{\scriptsize$\pm$1.58} & 80.56{\scriptsize$\pm$0.40} & -                          & 82.84{\scriptsize$\pm$1.08} & 81.17{\scriptsize$\pm$0.26} & 81.10{\scriptsize$\pm$0.74} & 80.61{\scriptsize$\pm$0.61} & - \\
        Shakespeare                            & 3.43{\scriptsize$\pm$0.01}  & 3.39{\scriptsize$\pm$0.04}  & 3.38{\scriptsize$\pm$0.03}  & 3.40{\scriptsize$\pm$0.01}  & 3.42{\scriptsize$\pm$0.01} & 3.86{\scriptsize$\pm$0.33}  & 3.54{\scriptsize$\pm$0.28}  & 3.45{\scriptsize$\pm$0.32}  & 3.80{\scriptsize$\pm$0.30}  & 3.60{\scriptsize$\pm$0.39} \\
        \bottomrule
    \end{tabular}
    }
    \vspace{2mm}
    \label{tab:fedoras_vs_random}
\end{table}

 \begin{figure}
    \centering
    \includegraphics[width=\linewidth]{images/equi_budget.pdf}
    \captionsetup{font=small,labelfont=bf}
    \caption{For each dataset, we randomly sample their search space and train these models end-to-end. We repeat this process (with new samples) three times, and overlap the scatter plots of different runs (i.e. red, green, blue). Then, for each tier we average the best performing models across each of the three runs and plot the result as a dashed horizontal line with grey area representing {\scriptsize $\pm$} standard deviation. The compute budget allocated to generate the data for each plot is equivalent to the cost of running {\tt FedorAS}{}, which also yields one model per tier.}
    \label{fig:equi_budget}
\end{figure}

\section{Convergence and Impact of Supernet Sampling}

 The proposed OPA aggregation scheme converges faster than an alternative aggregation method that does FedAvg of the updated supernets. We show this in Figure~\ref{fig:convergence}. When supernets from each client participating in the round get aggregated with OPA, the resulting supernet is consistently better in terms of validation accuracy than when aggregation is done with FedAvg. This difference becomes more evident as $B_{\Phi_{\text{comm}}}$ is reduced, i.e., as smaller portions of the supernet are sent to the clients. In Figure~\ref{fig:convergence} (d) we asses the feasibility of reducing the number of rounds and end the federated supernet training stage when validation accuracy reaches approximately 70\%. Those points correspond to 250 rounds and 150 rounds for the setting when 50\% and 25\% of the supernet is sent to the clients, respectively. The results that these settings yield show a significant loss (compared to their respective settings but over 500 rounds) and we therefore also run the same settings but when allowing for 50 additional runs. We observe a significant jump in per-tier model performance. %
 We leave as future work investigating alternative metrics to more accurately (but without incurring into heavy computational costs) measure the quality of the supernet at any given training iteration and leverage this to better inform an early stopping mechanism to further reduce communication costs.

\begin{figure*}
    \centering
    \begin{subfigure}[b]{0.495\textwidth}
        \centering
        \includegraphics[width=\textwidth]{images/convergence_full.pdf}
        \caption[Network2]%
        {{\small Sending whole of the supernet ($B_{\Phi_{\text{comm}}} = 22.3M$)}}
        \label{fig:mean and std of net14}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.495\textwidth}
        \centering
        \includegraphics[width=\textwidth]{images/convergence_50.pdf}
        \caption[]%
        {{\small Sending 50\% of the supernet ($B_{\Phi_{\text{comm}}} = 11.15M$)}}
        \label{fig:mean and std of net24}
    \end{subfigure}
    \vskip\baselineskip
    \begin{subfigure}[b]{0.495\textwidth}
        \centering
        \includegraphics[width=\textwidth]{images/convergence_25.pdf}
        \caption[]%
        {{\small Sending 25\% of the supernet ($B_{\Phi_{\text{comm}}} = 5.58M$)}}
        \label{fig:mean and std of net34}
    \end{subfigure}
    \hfill
    \begin{subfigure}[b]{0.495\textwidth}
        \centering
        \includegraphics[width=\textwidth]{images/convergence_mix.pdf}
        \caption[]%
        {{\small Impact of fewer rounds}}
        \label{fig:mean and std of net44}
    \end{subfigure}
    \captionsetup{font=small,labelfont=bf}
    \caption[ meh ]
    {\small Results using OPA compared to FedAvg in {\tt FedorAS}{} for CIFAR-10 non-IID ($\alpha=1.0$) with fixed hyperparameters. Each sub-plot contains two plots: first a scatter plot that visualises the federated supernet training (Sec.~\ref{sec:supernet_training}) in the first 5k steps and the per-tier search stage (Sec.~\ref{sec:model_validation_search}) the remaining steps; and, a bar plot that shows the average test accuracies of each tier after federated finetuning (Sec.~\ref{sec:finetuning_phase}). For (a)-(c), the training setups are identical with the exception of using either OPA or FedAvg. Both settings perform 500 rounds of federated supernet training using 10 clients per round (i.e. 5k steps). Every 10 rounds, the supernet is evaluated on the global validation set by randomly sampling paths and a dot is added to the plot. In the context of reduced communication budget, OPA largely outperforms FedAvg, requiring fewer federated rounds to reach the same validation accuracy. This difference is more noticeable when sending 50\% of the supernet (roughly corresponding to the size of a ResNet18 model). In (d) we measure the quality of the models found when the supernet training phase ends once 70\% validation accuracy is reached -- the highest reached by FedAvg in (a)-(c) -- and show competitive performance of models derived from OPA-aggregated supernets. By extending the training phase by 50 rounds, we observe large improvements in the quality of the final models.}
    \label{fig:convergence}
\end{figure*}

\section{Fairness} %

Up to this point, we have reported on the global test accuracy, which is a union over the test sets of the clients for all datasets but SpeechCommands, which defines a separate set of clients mutually exclusive to the ones participating in federated training.
In this section we want to measure how the ability of our system to aggregate knowledge across tiers during supernet training affects the fairness in participation.
We quantify this through variance and worst-case statistical measures of client performance on their respective test set. These test sets are considered to be following the same distribution (LDA or pre-split) of their train and validation sets.

Results are shown on Tab.~\ref{tab:fairness} for the different datasets offering per-client test data splits for all clients. It can be seen that {\tt FedorAS}{} leads consistently to better performance compared to end-to-end FL trained networks of the same architectures, where only eligible clients can train models.
With respect to the standard deviation of per client performance, we witness {\tt FedorAS}{} offering lower variance, except for the case of CIFAR-100. We consider this to be a consequence of our significantly higher accuracy. Similar behaviour, is also witnessed when we measure variance per tier of devices.
Last, we also showcase the worst-case result of a clients performance in the last column of the table.

  \begin{table}[h]
    \centering
    \small
    \captionsetup{font=small,labelfont=bf}
    \caption{Quantification of fairness with per client accuracy statistics, comparing end-to-end with {\tt FedorAS}'s performance. We report on the mean, standard deviation per tier and across tiers and minimum performance (min accuracy or max perplexity) across datasets with per-tier client test sets. Lower standard deviation is better.}
    \scalebox{0.8}{
    \begin{tabular}{lccccc}
        \toprule
        \multirow{2}{*}{Dataset} & Mode & \# Total    &  Perf.  & Perf. & Worst  \\
         & & Clients  &  (across tiers) & (per tier) & Perf.       \\
        \midrule
        {CIFAR-10}    & Fedoras & \multirow{2}{*}{100} & 81.36{\scriptsize$\pm$8.58} & (82.43{\scriptsize$\pm$7.87}, 81.41{\scriptsize$\pm$9.34}, 81.78{\scriptsize$\pm$8.49}, 79.80{\scriptsize$\pm$8.42}) & 47.00 \\
        \tiny{(Acc. (\%) $\uparrow$ is better)} & End2End & & 65.70 {\scriptsize$\pm$13.11} & (67.43{\scriptsize$\pm$12.29}, 66.10{\scriptsize$\pm$14.05}, 66.14{\scriptsize$\pm$13.02}, 63.12{\scriptsize$\pm$12.76}) & 11.00 \\
        \cmidrule{2-6}
        {CIFAR-100}   & Fedoras & \multirow{2}{*}{500} & 45.61{\scriptsize$\pm$13.15} & (44.46{\scriptsize$\pm$13.81}, 45.51{\scriptsize$\pm$12.28}, 45.19{\scriptsize$\pm$13.15}, 47.29{\scriptsize$\pm$13.15}) & 5.00 \\
        \tiny{(Acc. (\%) $\uparrow$ is better)} & End2End & & 31.08{\scriptsize$\pm$12.12} & (30.72{\scriptsize$\pm$13.16}, 30.65{\scriptsize$\pm$11.27}, 30.43{\scriptsize$\pm$11.25}, 32.52{\scriptsize$\pm$12.59}) & 0.00 \\
        \cmidrule{2-6}
        {Shakespeare}  & Fedoras & \multirow{2}{*}{715} & 2.93{\scriptsize$\pm$1.01}
        & (2.93{\scriptsize$\pm$0.97}, 2.94{\scriptsize$\pm$0.99}, 2.88{\scriptsize$\pm$1.03}, 2.98{\scriptsize$\pm$1.04}, 2.94{\scriptsize$\pm$1.01})  & 8.43 \\
        \tiny{(Perplexity $\downarrow$ is better)} & End2End & & 3.07{\scriptsize$\pm$1.10} & (3.07{\scriptsize$\pm$1.05}, 3.08{\scriptsize$\pm$1.10}, 3.02{\scriptsize$\pm$1.13}, 3.11{\scriptsize$\pm$1.13}, 3.07{\scriptsize$\pm$1.10}) & 8.36 \\
        \bottomrule
    \end{tabular}
    }
    \vspace{2mm}
    \label{tab:fairness}
\end{table}

An in-depth view of how each client behaves is depicted in Fig.~\ref{fig:cifar10-fairness} for CIFAR-10, where we show performance of Fedoras vs. end-to-end FL-trained models per client.

\begin{figure}[t]
    \centering
    \includegraphics[trim={4cm, 0, 4cm, 0}, clip, width=\linewidth]{images/cifar10_per_clients_stacked.pdf}
    \captionsetup{font=small,labelfont=bf}
    \caption{Accuracy per client of {\tt FedorAS}{} vs. end-to-end FL-trained models (same architecture) on CIFAR-10. Accuracy is quantified on each client's dataset from the model associated with that device's tier and error bars represent the standard deviation of accuracy between different runs. Across runs, the allocation of data to clients does not change. Ordered by ascending end-to-end accuracy.}
    \label{fig:cifar10-fairness}
\end{figure}

\subsection{Experimental Setup}

\noindent
\textbf{Datasets.} Datasets are summarised in Tab.~\ref{tab:datasets}.
We partition CIFAR-10/100 following the Latent Dirichlet Allocation (LDA) partitioning~\cite{hsu2019measuring}, with each client receiving approximately equisized training sets. For CIFAR-10, we consider
$\alpha \in \{1000, 1.0, 0.1\}$ configurations, while for CIFAR-100, we adopt
$\alpha=0.1$ as in~\cite{reddi2020adaptive}. The remaining datasets come naturally partitioned.

\begin{minipage}{0.45\textwidth}
    \begin{table}[H]
        \begin{subtable}[t]{\textwidth}
        \centering
        \setlength{\tabcolsep}{1pt}
        \small
        \scalebox{0.76}{
            \begin{tabular}{lccrll}
                \toprule
                \multirow{2}{*}{\textbf{Dataset}}     & \textbf{Search}    & \textbf{Number} & \textbf{Number} & \multirow{2}{*}{\textbf{Target Task}} \\
                          &       \textbf{Space}               & \textbf{Clients}               & \textbf{Examples}     \\
                \midrule
                CIFAR10     & CNN-L & $100$         & $50K$      & Image classification \\
                CIFAR100     & CNN-L & $500$         & $50K$      & Image classification \\
                Speech Commands   & CNN-S & $2,112$         & $105.8K$      & Keyword Spotting \\
                Shakespeare & RNN      & $715$          & $38K$      & Next char. prediction \\
                \bottomrule
            \end{tabular}
        }
        \end{subtable}%
        \vspace{5mm}
        \captionsetup{font=small,labelfont=bf}
        \caption{High level overview of the datasets considered for evaluating {\tt FedorAS}. More details about datasets and search spaces in the Appendix.}
        \label{tab:datasets}
    \end{table}
\end{minipage}
\hfill
\begin{minipage}{0.50\textwidth}
    \begin{figure}[H]
        \centering
        \vspace{-7mm}
        \includegraphics[width=\textwidth]{images/fedoras_plot.pdf}
        \captionsetup{font=small,labelfont=bf}
        \vspace{-5mm}
        \caption{{\tt FedorAS}{} surpasses the performance of models across tiers, even for unrestricted settings. CIFAR-10 (non-IID, $\alpha$=1.0)}
        \label{fig:fedoras_plot}
    \end{figure}
\end{minipage}

\noindent
\textbf{Search space.}
The adopted search spaces are specific to the dataset and task at hand, both in terms of size and type of operators. In general, we assume our highest tier of model sizes to coincide with a typical network footprint used for that task in the FL literature (e.g. ResNet-18 for CIFAR-10). Nevertheless, there may be operators in the space that cannot be sampled in some tiers due to their footprint.
{\tt FedorAS}{} sets the communication budget $B_{\Phi_{\text{comm}}}$ to be half the size of the supernet.

\noindent
\textbf{Clusters definition.}
In our experiments, we cluster clients based on the amount of operations they can sustain per round of training (\#FLOPs), for simplicity. Other resources (e.g. \#parameters, energy consumption) can be used in place or in conjunction with FLOPs. Unless otherwise stated, clusters are equisized in terms of \#devices associated with each.
We assume that the largest model in the search space is the maximal size that any of the clients can handle. The minimum cost is set to the fixed cost of the network (i.e. cost of non-searchable components). Given this range, and number of clusters $C$, we define the first and last clusters at $prct_l, prct_r$ percentiles of a randomly sampled set of models from the search space and linearly scale the $C-2$ clusters in-between.

\noindent
\textbf{Implementation.}
{\tt FedorAS}~was implemented on top of the \texttt{Flower} (v0.18)~\cite{beutel2020flower} framework and PyTorch (v1.11.0) \cite{pytorch2019}. We run all our experiments on a private cloud cluster in a simulated manner, across four iterations each and report averages and standard deviations.

\subsection{Performance evaluation}\label{sec:fedoras_vs_sota}

\vspace{-0.4cm}
\begin{table*}[!t]
    \centering
    \captionsetup{font=small,labelfont=bf}
    \caption{Comparison of selected models obtained with {\tt FedorAS}{} with relevant state-of-the-art models from the literature. Perf. -- performance, S. -- searching, T. -- training.}
    \scalebox{0.625}{
        \begin{tabular}{llcccccccl}
            \toprule
            \small
            & \textbf{Method} & \textbf{Perf.} & \textbf{FLOPs (M)} & \textbf{Param. (M)} & \textbf{S. Cost$^{\dagger}$} & \textbf{T. Cost$^{\dagger}$} & \textbf{Setting}  & \textbf{Notes} \\
            \midrule

            \multicolumn{8}{@{}l}{\textbf{CIFAR-10}} \\
            \cdashlinelr{1-9}
            1 & FjORD {\scriptsize $p=1.0$}\cite{fjord}                               & 84.69                                & 264.4{\scriptsize$\pm$191.87}          & 11.4           & -                               &  1$\times$10$\times$500                   & \textit{c.device}              &      non-LDA \\
            2 & FjORD {\scriptsize $p=1.0$ (KD)}\cite{fjord}                         & 85.56                                & 264.4{\scriptsize$\pm$191.87}          & 11.4           & -                               &  1$\times$10$\times$500                & \textit{c.device}                  &       non-LDA \\
            3 & FjORD{\scriptsize $p=0.4$ (KD)}\cite{fjord}                          & 84.11                                & 57.0{\scriptsize$\pm$34.0}           & 2.0             & -                               &  1$\times$10$\times$500                  & \textit{c.device}                &       non-LDA \\
            4 & SPIDER~\cite{Mushtaq2021-eq}                                   & 92.00{\scriptsize$\pm$2.00}                    & 62.0{\scriptsize$\pm$?}       & 0.3           & ?$\times$8$\times$(60+260)                               &  ?$\times$8$\times$680                             & \textit{c.silo}     & $\alpha=0.2$, 8 clients \\
            5 & FedNAS~\cite{He2020-bp}                                        & 90.70{\scriptsize$\pm$2.40}                    & -           & 4.0            & 5$\times$20$\times$50                               &  ?                             & \textit{c.silo}     & $\alpha=0.5$, 20 clients \\
            6 & {\tt FedorAS}{} {\scriptsize best}                                      & 86.88{\scriptsize$\pm$0.32}                     & 148.0         & 2.74        & \multirow{2}{*}{50$\times$10$\times$500}        & \multirow{2}{*}{1$\times$6$\times$100}              & \multirow{2}{*}{\textit{c.device}}  & \multirow{2}{*}{$\alpha=1.0$, 100 clients} \\
            7 & {\tt FedorAS}{} {\scriptsize avg.}                                      & 86.30{\scriptsize$\pm$0.41}                    & 183.0{\scriptsize$\pm$25.2}         & 2.81{\scriptsize$\pm$0.32}        &                           &       &                                     & \\
            \midrule

            \multicolumn{8}{@{}l}{\textbf{CIFAR-100} \textit{(500 clients, $\alpha=0.1$)}} \\
            \cdashlinelr{1-9}
            9 & ResNet-18 {\scriptsize FedAvg}\cite{reddi2020adaptive}         & 44.70                             & 1,112          & 11.4           & -                               &  1$\times$10$\times$4000                                 & \multirow{5}{*}{\textit{c.device}} &       GroupNorm~\cite{Wu_2018_ECCV} \\
            10 & ResNet-18 {\scriptsize FedAdam}\cite{reddi2020adaptive}        & 52.50                             & 1,112          & 11.4           & -                               &  1$\times$10$\times$4000                                 &  &       GroupNorm~\cite{Wu_2018_ECCV} \\
            11 & {\tt FedorAS}{} {\scriptsize End2End}                                   & 63.93                            & 89.4         & 1.62       & -                                  & 1$\times$10$\times$4000                              &  & FedAvg aggr. \\
            12 & {\tt FedorAS}{} {\scriptsize best}                                      & 46.02       & 162.0         & 2.27        & \multirow{2}{*}{25$\times$10$\times$750}        & \multirow{2}{*}{1$\times$6$\times$100}              &                            &                   \\
            13 & {\tt FedorAS}{} {\scriptsize avg.}                                      & 45.84{\scriptsize$\pm$0.18}       & 214{\scriptsize$\pm$7.6}         & 2.32{\scriptsize$\pm$0.34}        &                           &        &                                     & \\
            \midrule

            \multicolumn{8}{@{}l}{\textbf{Speech Commands}} \\
            \cdashlinelr{1-9}
            14 & Oort~\cite{oort}                                               & 62.20                            & 4,761$^{\ddagger}$            & 21.29      & -                               & >50h                      & \multirow{4}{*}{\textit{c.device}} & ResNet34, FedYoGi\cite{reddi2020adaptive}\\
            15 & PyramidFL~\cite{lipyramidfl}                                   & 63.84                            & 4,761$^{\ddagger}$            & 21.29      & -                               & 10.3h                      &  & ResNet34, FedProx\cite{reddi2020adaptive}\\
            16 & {\tt FedorAS}{} {\scriptsize best}                                      & 70.10       & 10.0         & 0.63        & \multirow{2}{*}{50$\times$21$\times$750}        & \multirow{2}{*}{1$\times$21$\times$100}              &   & \multirow{2}{*}{--} \\
            17 & {\tt FedorAS}{} {\scriptsize avg.}                                      & 68.49{\scriptsize$\pm$1.47}       & 11.87{\scriptsize$\pm$0.66}         & 0.71{\scriptsize$\pm$0.06}        &                                 & &                                       & \\
            \midrule

            \multicolumn{8}{@{}l}{\textbf{Shakespeare}} \\
            \cdashlinelr{1-9}
            18 & RNN{\scriptsize $p=1.0$}\cite{fjord}                            & 3.98                                & 34.6          & 0.214           & -                               &  1$\times$10$\times$500                                & \multirow{5}{*}{\textit{c.device}}   &       -- \\
            19 & RNN {\scriptsize $p=1.0$ (KD)}\cite{fjord}                      & 3.87                                & 34.6         & 0.214           & -                               &  1$\times$10$\times$500                            &       &       -- \\
            20 & RNN {\scriptsize $p=0.4$ (KD)}\cite{fjord}                      & 3.91                                & 6.4          & 0.040           & -                               &  1$\times$10$\times$500                             &       &       -- \\
            21 & {\tt FedorAS}{} {\scriptsize best}                                      & 3.34       & 13.4         & 0.167        & \multirow{2}{*}{5$\times$16$\times$500}        & \multirow{2}{*}{1$\times$16$\times$100}             &  & \multirow{2}{*}{--} \\
            22 & {\tt FedorAS}{} {\scriptsize avg.}                                      & 3.38{\scriptsize$\pm$0.04}       & 11.75{\scriptsize$\pm$0.43}         & 0.146{\scriptsize$\pm$0.005}        &                                 &                                     & \\

            \bottomrule \addlinespace[4pt]
            \multicolumn{8}{l}{$\dagger$ Reported as $\text{local\_epochs} \times \text{clients\_per\_round} \times \text{rounds}$ or as wall time (hrs), $\ddagger$Assumed 40$\times$51 input MFCC features\cite{zhang2018hello}}\\
        \end{tabular}
    }
    \label{tab:comparison}
    \vspace{-0.1cm}
\end{table*}

We start the evaluation by painting the landscape of performance in the cross-silo and cross-device FL settings and place {\tt FedorAS}{} in the context of resource-efficient federated NAS in the latter setting. Specifically, we draw from the results of several baselines \cite{fjord,He2020-bp,Mushtaq2021-eq,reddi2020adaptive,lipyramidfl,oort} and annotate their evaluation settings and reported results in Tab.~\ref{tab:comparison} across tasks.
It can be seen that, in general, cross-silo settings are imposing less restrictive requirements\footnote{Works assume that all models can (non-heterogeneous) and do (full-participation) run everywhere.} in training. Concretely, they operate under full client participation and are not restricted by the capabilities of the clients (non-heterogeneous), be it in the form of computation, memory or communication bandwidth resources. Moreover, their scalability in terms of number of total participating clients is significantly lower compared to the cross-device setting, spanning for example 8 \cite{Mushtaq2021-eq} or 20 clients \cite{He2020-bp}. Under these settings, one expectedly witnesses higher accuracies (e.g. rows 4-7).

On the other hand, compared to systems that come from heterogeneous FL \cite{fjord}, we see that the more degrees of freedom exposed in our setting result in better results, for Shakespeare and even under non-IID settings for CIFAR-10 (rows 1-3 vs. 6-7). The same is witnessed with client selection baselines \cite{oort,lipyramidfl} (rows 14-17). Last, compared normal FL baselines, Fedoras performs on par with FedAvg (rows 9, 13-14) and when the architectures are trained end-to-end in a federated manner, it surpasses even the FedAdam baseline (rows 9-12).

\subsection{Supernet weight-sharing with FedorAS}\label{sec:supernet_vs_random_init}

In this section, we evaluate the impact of weight sharing through {\tt FedorAS}' supernet. To this end, we compare the performance of {\tt FedorAS}' models to the same architectures models, but trained end-to-end in a federated manner, across all four datasets. Results are presented in Tab.~\ref{tab:results}.

\noindent
\textbf{Training on the same task.}
The standard setting across datasets is that we first train with {\tt FedorAS}{} across the three stages described in Sec.~\ref{sec:fedoras}. Subsequently, we take the output architectures from our system, randomly initialise them and train end-to-end in a federated manner, where each model trains across all eligible clients (tier and above).
With this comparison, we aim to showcase that {\tt FedorAS}{} not only comes up with better architectures, but it also effectively transfers knowledge between tiers through its supernet structure and OPA aggregation scheme.
Indeed, we can see in all cases models belonging to the most restricted tiers (least amount of capable clients) benefit from being initialised from a supernet.
The improvement is further magnified as non-IID-ness increases, leading to increases of up to 15 p.p. over random initialisation.
Interestingly, we also see that in many cases low-tier models seem to benefit -- we attribute this to the fact that, by being a part of the supernet, their weights are also updated in a context of much larger models which allows them to eventually reach better minima.
Regardless of what initialisation is used, we can see that models found by {\tt FedorAS}{} present state-of-the-art performance, as already shown in Table~\ref{tab:comparison}.

\noindent
\textbf{Training on different tasks.}
Specifically for CIFAR-100 and SpeechCommands, we also create a second scenario, where not all clients train in the same domain of labels. For CIFAR-100, there are 20 superclasses that are coarse-grained categories of the 100 standard labels. In contrast, for SpeechCommands, there are 12 and 35-class label sets, with the latter annotating the ``other'' class more specifically. In both cases, we assume a non-uniform distribution of clients to clusters, assuming that most data (75-80\%) reside on lower-tier devices, whereas the two higher tier-devices (holding 20-25\% of data) can train on the fine-grained label set. Our aim is to test whether few data allocation on the high tier devices can benefit from the knowledge and feature extraction learned from the coarse-grained label set.
We adopt two different setups. For CIFAR-100, we train both tasks simultaneously, having essentially two distinct linear layers across tiers (1,2) and (3,4). We call this setup multi-task. On the other hand, for SpeechCommands, we train all clients on the same coarse domain and subsequently transfer learn a fine-grained 35 classes linear layer for the client of tiers (3,4). We fine-tune the linear layer keeping the rest of the network frozen for 25 epochs and then allow for finetuning of the whole network for another 100.

In both cases {\tt FedorAS}{} learns better models and transfer knowledge from the low-tiers to high-tiers and vice-versa through weight sharing of the supernet. Indicatively, for CIFAR-100 we are able to achieve $+14.91$ p.p (percentage points) of accuracy compared to training the same architectures end-to-end on eligible devices. Similarly, we achieve $+1.6$ p.p higher accuracy for SpeechCommands.

\vspace{0.3cm}
\subsection{Ablation study}
\vspace{0.2cm}
\label{sec:ablation}

Here we measure the contribution of specific components of {\tt FedorAS}{} to the quality and performance of these models. To this direction, we firstly compare the performance of our system's aggregation compared to naive averaging. Subsequently, we measure the impact of sending smaller subspaces to clients for supernet training. We provide indicative results CIFAR-10 (non-IID, $\alpha$=1.0).

\begin{table*}[!t]
    \centering
    \captionsetup{font=small,labelfont=bf}
    \caption{Models discovered by {\tt FedorAS}{} benefit from weight sharing across tiers compared to models using the same architecture but trained end-to-end on clients that support them. Models derived from a {\tt FedorAS}{} supernet outperform their baselines by large margins in most cases. First part: accuracy, second part: perplexity.}
    \scalebox{0.685}{
        \begin{tabular}{cccccccccc}
            \toprule
            \small
            \textbf{Dataset} & \textbf{Clients} & \textbf{Setting} & \textbf{Partitioning} & \textbf{Mode}  & \textbf{Classes} & \textbf{Tier 1} & \textbf{Tier 2} & \textbf{Tier 3} & \textbf{Tier 4}\\
            \midrule
            \multirow{7}{*}{CIFAR-10} & \multirow{7}{*}{100} & \multirow{7}{*}{Standard} & \multirow{2}{*}{$\text{IID}_{\alpha=1000}$}          & {\tt FedorAS}{}   &   \multirow{2}{*}{10}  & 89.40{\scriptsize$\pm$0.19} & 89.60{\scriptsize$\pm$0.15} & 89.64{\scriptsize$\pm$0.22} & 89.24{\scriptsize$\pm$0.29} \\
                                      &                     &                    &                                                              & End2End   &                & 89.05{\scriptsize$\pm$0.17} & 87.84{\scriptsize$\pm$0.38} & 86.18{\scriptsize$\pm$0.38} & 81.27{\scriptsize$\pm$0.81} \\
            \cmidrule{4-10}
                                      &                     &                    &\multirow{2}{*}{$\text{non-IID}_{\alpha=1.0}$}                   & {\tt FedorAS}{} & \multirow{2}{*}{10}                 & 85.99{\scriptsize$\pm$0.13} & 86.30{\scriptsize$\pm$0.41} & 86.34{\scriptsize$\pm$0.19} & 85.58{\scriptsize$\pm$0.55} \\
                                      &                     &                    &                                                              & End2End   &                & 87.12{\scriptsize$\pm$0.44} & 86.29{\scriptsize$\pm$0.86} & 85.10{\scriptsize$\pm$0.44} & 80.10{\scriptsize$\pm$1.92} \\
            \cmidrule{4-10}
                                      &                     &                    & \multirow{2}{*}{$\text{non-IID}_{\alpha=0.1}$}                  & {\tt FedorAS}{} &  \multirow{2}{*}{10}                & 81.01{\scriptsize$\pm$0.46} & 81.53{\scriptsize$\pm$0.29} & 80.64{\scriptsize$\pm$0.66} & 80.85{\scriptsize$\pm$0.28} \\
                                      &                     &                    &                                                              & End2End  &                 & 70.61{\scriptsize$\pm$2.16} & 70.30{\scriptsize$\pm$1.90} & 68.29{\scriptsize$\pm$0.49} & 64.87{\scriptsize$\pm$1.48}\\
            \midrule
            \multirow{5}{*}{CIFAR-100} & \multirow{5}{*}{500} & \multirow{2}{*}{Standard} & \multirow{2}{*}{$\text{non-IID}_{\alpha=0.1}$}         & {\tt FedorAS}{} &   \multirow{2}{*}{100}                  & 45.25{\scriptsize$\pm$0.13} & 45.84{\scriptsize$\pm$0.18} & 45.42{\scriptsize$\pm$0.39} & 45.07{\scriptsize$\pm$0.71} \\
                                     &                     &                   &                                                                & End2End  &                 & 36.30{\scriptsize$\pm$0.96} & 39.26{\scriptsize$\pm$1.21} & 39.06{\scriptsize$\pm$1.32} & 36.77{\scriptsize$\pm$1.32} \\
            \cmidrule{3-10}
                                     &                          & \multirow{2}{*}{Multi-task}  &  \multirow{2}{*}{$\text{non-IID}_{\alpha=0.1}$}     & {\tt FedorAS}{} & \multirow{2}{*}{$20\rightarrow100$}                  & 57.93{\scriptsize$\pm$0.31} & 57.86{\scriptsize$\pm$0.7} & 38.63{\scriptsize$\pm$0.74} & 37.68{\scriptsize$\pm$0.73} \\
                                     &                          &                     &                                                         & End2End &  & 41.64{\scriptsize$\pm$0.26} & 42.45{\scriptsize$\pm$0.58} & 27.11{\scriptsize$\pm$0.64} & 21.28{\scriptsize$\pm$1.26} \\

            \midrule
            \multirow{5}{*}{\parbox{1.75cm}{Speech\\ Commands}} & \multirow{5}{*}{2112} & \multirow{2}{*}{Standard} & \multirow{2}{*}{\textit{given}}   & {\tt FedorAS}{}  &  \multirow{2}{*}{12}       & 80.19{\scriptsize$\pm$1.78} & 80.47{\scriptsize$\pm$1.69} & 81.0{\scriptsize$\pm$1.58}  & 80.56{\scriptsize$\pm$0.40} \\
                                      &                         &                   &                                                                   & End2End  &         & 81.92{\scriptsize$\pm$1.32} & 79.94{\scriptsize$\pm$0.84} & 78.57{\scriptsize$\pm$1.42} & 79.36{\scriptsize$\pm$1.67} \\
            \cmidrule{3-10}
                                     &                          & \multirow{2}{*}{Transfer}  & \multirow{2}{*}{\textit{given}}                  & {\tt FedorAS}{}   & \multirow{2}{*}{$12\rightarrow35$}               & 67.06{\scriptsize$\pm$2.12}$^{\dagger}$ & 66.87{\scriptsize$\pm$1.85}$^{\dagger}$ & 67.65{\scriptsize$\pm$1.59} & 68.49{\scriptsize$\pm$1.47} \\
                                     &                          &                     &                                                         & End2End  &                 & 64.99{\scriptsize$\pm$1.41}$^{\dagger}$ & 65.03{\scriptsize$\pm$0.69}$^{\dagger}$ & 66.55{\scriptsize$\pm$2.2}  & 66.84{\scriptsize$\pm$0.87} \\
        \end{tabular}
    }

    \vspace{1mm}

    \scalebox{0.70}{
            \begin{tabular}{ccccccccccc}
            \toprule
            \small
            \textbf{Dataset} & \textbf{Clients} & \textbf{Setting} & \textbf{Partitioning} & \textbf{Mode}  & \textbf{Classes} & \textbf{Tier 1} & \textbf{Tier 2} & \textbf{Tier 3} & \textbf{Tier 4} & \textbf{Tier 5}\\
            \midrule
            \multirow{2}{*}{Shakespeare} & \multirow{2}{*}{715} & \multirow{2}{*}{Standard} & \multirow{2}{*}{\textit{given}}                   & {\tt FedorAS}{}                  & \multirow{2}{*}{90} & 3.43{\scriptsize$\pm$0.01}  & 3.39{\scriptsize$\pm$0.04}  & 3.38{\scriptsize$\pm$0.03}  & 3.40{\scriptsize$\pm$0.01}   & 3.42{\scriptsize$\pm$0.01} \\
                                      &                       &                    &                                                            & End2End                  &  & 3.44{\scriptsize$\pm$0.03}  & 3.50{\scriptsize$\pm$0.02}   & 3.47{\scriptsize$\pm$0.08}  & 3.52{\scriptsize$\pm$0.07}  & 3.60{\scriptsize$\pm$0.04} \\
            \bottomrule \addlinespace[4pt]
            \multicolumn{8}{l}{$\dagger$ Trained only on clients belonging to Tier 3 and 4.}
        \end{tabular}
    }
    \label{tab:results}
    \vspace{-0.6cm}
\end{table*}

\textbf{OPA vs. naive averaging.}
OPA compared to naive FedAvg leeds to increased accuracy by +1.2, +0.9, +1.01 and +0.9 p.p. for tiers 1-4, respectively.

\textbf{Subspace sampling size.}
{\tt FedorAS}{} samples the supernet before communicating it to a client. For CIFAR-10 ($\alpha=1.0$) and when $B_{\Phi_{\text{comm}}}$ is set to 22.3M parameters (i.e. allowing sending whole supernet), {\tt FedorAS}{} yields 85.475\%, 86.725\%, 86.5\% average accuracies across tiers for full, $1/2$ and $1/4$ of the size of the search space, respectively.
Thus, not only is the overhead of communication reduced with {\tt FedorAS}, but convergence to the same level of accuracy is faster. We hypothesise that this is due a better balance in the exploration vs. exploitation trade-off.

redpajama_set_name: RedPajamaArXiv


text:
Christian Meier (* 16. Februar 1929 in Stolp/Pommern) ist ein deutscher Althistoriker. Meier lehrte von 1981 bis zu seiner Emeritierung im Jahr 1997 als Professor für Alte Geschichte an der Ludwig-Maximilians-Universität München.

Leben
Der Sohn eines Landwirts und Enkel von Paul Jonas Meier besuchte Gymnasien in Stettin, Rostock und Hamburg. 1948 legte Meier in Hamburg sein Abitur ab und studierte anschließend Geschichte, Klassische Philologie und Römisches Recht an der Universität Rostock, wo er durch Ernst Hohl für die Alte Geschichte begeistert wurde, bevor er an die Universität Heidelberg wechselte. 1956 wurde er dort bei Hans Schaefer promoviert. Seine Habilitation erfolgte 1963 in Frankfurt am Main. Dort war er Schüler von Hermann Strasburger und Matthias Gelzer. Ab 1964 war er Privatdozent in Freiburg i. B.; danach folgten Professuren für Alte Geschichte in Basel (1966 bis 1968), Köln (1968 bis 1973), wiederum Basel (1973 bis 1976) und Bochum (1976 bis 1981). 1981 wurde er schließlich Nachfolger Siegfried Lauffers auf dem Lehrstuhl für "Alte Geschichte unter besonderer Berücksichtigung der Sozial- und Wirtschaftsgeschichte" in München, den er bis zu seiner Emeritierung 1997 innehatte. Im akademischen Jahr 1984/1985 war Meier Fellow am Wissenschaftskolleg zu Berlin. Christian Meier ist mit einer Germanistin verheiratet.

Wirken
Meier ist einer der bekanntesten Historiker Deutschlands seiner Generation, zumal er sich mit einigen seiner Werke auch an die breite Öffentlichkeit wandte. Als bis heute enorm einflussreich erwies sich bereits seine Habilitationsschrift Res publica amissa. Weit über die Fachkreise hinaus bekannt geworden ist er durch seine beiden Bücher Caesar und Athen. Ein Neubeginn der Weltgeschichte. Seine 1982 veröffentlichte Caesar Biographie gilt in der Fachwelt als Meilenstein auf dem Weg zu einer Rehabilitierung der Biographie innerhalb der deutschen Geschichtswissenschaft.

Das Besondere an seiner Herangehensweise war die schon von seinem akademischen Lehrer Hans Schaefer und dessen Schule proklamierte Abwendung von der seit der Renaissance weithin unreflektiert eingenommenen klassizistischen und identifikatorischen Perspektive, die jahrhundertelang auf der Vorstellung von einer auf Wesensverwandtschaft und ungebrochenem Traditionszusammenhang beruhenden unmittelbaren Zugänglichkeit der antiken Kultur für die Menschen der europäischen Moderne beruht hatte. Die Antike erschien Meier vielmehr als das "nächste Fremde", dessen Verstehen von Grund auf neu zu erarbeiten und nur in einem hermeneutischen Vorgehen zu gewinnen sei. Schwerpunkte seiner Forschung sind das klassische Athen und die späte Römische Republik, deren Krisenzeit und Niedergang er in Res publica amissa mit einer berühmten Formulierung als eine "Krise ohne Alternative" beschrieb. Damit war nicht etwa gemeint, dass es keine Alternative zur Krise der Republik gegeben hätte, sondern dass die damaligen Akteure zumindest anfangs über keinen alternativen Entwurf zur bestehenden, in eine Krise geratenen politischen Ordnung verfügt hätten. Die Folge sei gewesen, dass sie diese zerstörten, ohne dies zu beabsichtigen und ohne gezielt auf die Etablierung einer neuen politischen Ordnung hinzuarbeiten. Konsens habe vielmehr darüber geherrscht, die unübersehbaren Krisenphänomene als Verfall der überkommenen Ordnung zu deuten, dem mit dem Ziel entgegenzutreten sei, diese wiederherzustellen. Insofern war für Meier die Anwendung des Revolutionsbegriffs im Sinne einer Revolutionstheorie auf die späte Republik nicht sinnvoll. Folgerichtig habe Augustus seine Neuordnung, die auf die Etablierung einer Monokratie hinauslief, nur als Wiederherstellung der alten Ordnung (res publica restituta) präsentieren können.

In seiner Rezension zu Meiers Die Entstehung des Politischen bei den Griechen von 1980 stellte Ernst Vollrath heraus, dass es Meier in dieser Sammlung von Aufsätzen im Kern um die entscheidenden systematischen Fragen gehe: "Was eigentlich ist das Politische? Was ist sein Begriff? Wie haben die Griechen als die ersten, die dieses neue Feld menschlichen Daseins bewusst betreten haben, das Politische und seinen Begriff erfahren und gefasst?" Von Hannah Arendt und Dolf Sternberger ausgehend und in Auseinandersetzung mit Carl Schmitts Begriff der Dezision, sehe Meier das Politische phänomenologisch-historisch als eine besondere Art des Wissens und Handelns, das er als "Könnensbewusstsein" bezeichnet, das Wissen um Möglichkeiten und Spielräume im Unterschied zu den automatischen Abläufen der als unverfügbar verstandenen vorgegebenen Ordnung des Gewohnheitsrechts (Nomos). Der Nomos wird politisiert und verwandelt sich so in den Begriff der Regelungen, die sich Menschen gemeinschaftlich selber geben, damit der ursprüngliche Sinn der Regelungen nicht korrumpiert wird. Meier zeige sein Politikverständnis besonders an den Eumeniden des Aischylos.

Beim Historikerstreit war Meier bereit, offensiv Stellung zu beziehen, um Positionen zu klären. Als Historiker hat er den Anspruch, erzählerische Geschichtsschreibung zu betreiben. Nicht alles kann erzählt werden, und oft sind es eher Wahrscheinlichkeiten als Tatsachen (vgl. dazu Meier: Athen, Taschenbuchausgabe 1997, S. 692 f.). Als leidenschaftlicher Vertreter und Promotor einer am Angelsächsischen geschulten Geschichtsschreibung verbindet er hohes wissenschaftliches Niveau mit literarischen Ansprüchen.

Meier blickt immer wieder über den Tellerrand seines Fachs. So beschäftigte er sich mit der modernen Demokratie und der Politik der Bundesrepublik Deutschland (vor allem im Zuge der Wiedervereinigung).

2015 schenkte Meier sein Privatarchiv (Manuskripte und Korrespondenzen) dem Deutschen Literaturarchiv Marbach, darunter Briefwechsel mit Carl Schmitt und Richard von Weizsäcker.

Preise und Auszeichnungen
1998 wurde er für seine eigene Beredsamkeit mit dem Cicero-Rednerpreis ausgezeichnet. Beim Thema Rechtschreibreform äußerte sich Meier in seiner Rolle als Präsident der Deutschen Akademie für Sprache und Dichtung engagiert und kritisch. Für seinen Einsatz für die deutsche Sprache überreichte Eberhard Schöck Christian Meier 2003 den Jacob-Grimm-Preis. 2009 und 2015 verliehen die Universitäten Salzburg und Bern Meier die Ehrendoktorwürde.

2004 erhielt er den Ernst-Hellmut-Vits-Preis der Westfälischen Wilhelms-Universität in Münster (Westfalen). Am 2. März 2006 wurde Meier vom österreichischen Bundespräsidenten Heinz Fischer das Österreichische Ehrenzeichen für Wissenschaft und Kunst überreicht. 2007 wurde Meier mit dem Reuchlinpreis ausgezeichnet, 2009 mit der Lichtenberg-Medaille, 2014 mit dem Bayerischen Maximiliansorden für Wissenschaft und Kunst.

Mitgliedschaften und Funktionen
Meier war von 1980 bis 1988 Vorsitzender des Verbandes der Historiker Deutschlands und von 1981 bis 1995 Kurator des Historischen Kollegs in München. Meier ist zudem Mitbegründer der Berlin-Brandenburgischen Akademie der Wissenschaften und war von 1996 bis 2002 Präsident der Deutschen Akademie für Sprache und Dichtung in Darmstadt. Ferner gehört er der Norwegischen Akademie der Wissenschaften an. Seit 1989 ist er ordentliches Mitglied der Academia Europaea.

Schriften (Auswahl)
Viele Werke Meiers wurden in andere Sprachen übersetzt und erschienen in mehreren Neuauflagen. Der Siedler-Verlag brachte ihm zu Ehren seine Caesar-Biographie und sein Athen-Buch in einer Neuausgabe heraus. Meier verfasste zahlreiche Lexika- und Zeitschriftenartikel. Außerdem ist er Herausgeber mehrerer Sammelwerke und wissenschaftlicher Beirat der Zeitschrift Damals.

 Res publica amissa. Eine Studie zu Verfassung und Geschichte der späten römischen Republik. Steiner, Wiesbaden 1966 (Zugleich: Frankfurt am Main, Universität, Habilitations-Schrift, 1962).
 Entstehung des Begriffs Demokratie. Vier Prolegomena zu einer historischen Theorie. (= edition suhrkamp, Band 387) Suhrkamp, Frankfurt am Main 1970.
 Die Entstehung des Politischen bei den Griechen. Suhrkamp, Frankfurt am Main 1980, ISBN 3-518-07505-5.
 Die Ohnmacht des allmächtigen Dictators Caesar. Drei biographische Skizzen (= Edition Suhrkamp. Band 1038 = Neue Folge, Band 38). Suhrkamp, Frankfurt am in 1980, ISBN 3-518-11038-1.
 Caesar. Severin und Siedler, Berlin 1982, ISBN 3-88680-027-X.
 Die politische Kunst der griechischen Tragödie. Beck, München 1988, ISBN 3-406-33392-3; 2., erw. Auflage Beck, München 2022.
 mit Paul Veyne: Kannten die Griechen die Demokratie? 2 Studien (= Kleine kulturwissenschaftliche Bibliothek, Band 2). Wagenbach, Berlin 1988, ISBN 3-8031-5102-3.
 Deutsche Einheit als Herausforderung. Welche Fundamente für welche Republik? Hanser, München u. a. 1990, ISBN 3-446-16124-4.
 Die Nation, die keine sein will. Hanser, München und andere 1991, ISBN 3-446-16398-0.
Die Rolle des Krieges im klassischen Athen (= Schriften des Historischen Kollegs, Dokumentationen Band 6). München 1991 (Digitalisat).
 Athen. Ein Neubeginn der Weltgeschichte. Siedler, Berlin 1993, ISBN 3-88680-128-4.
 Die Parlamentarische Demokratie. Hanser, München und andere 1999, ISBN 3-446-19656-0.
Von Athen bis Auschwitz. Betrachtungen zur Lage der Geschichte (= Krupp-Vorlesungen zu Politik und Geschichte am Kulturwissenschaftlichen Institut im Wissenschaftszentrum Nordrhein-Westfalen, Band 2). Beck, München 2002, ISBN 3-406-48982-6.
 als Herausgeber: Sprache in Not? Zur Lage des heutigen Deutsch. Wallstein-Verlag, Göttingen 1999, ISBN 3-89244-341-6.
 Kultur, um der Freiheit willen. Griechische Anfänge – Anfang Europas? Siedler, München 2009, ISBN 978-3-88680-923-3 (Bildet den ersten Teil von Meiers Beitrag für die Siedler Geschichte Europas.).
 Das Gebot zu vergessen und die Unabweisbarkeit des Erinnerns. Vom öffentlichen Umgang mit schlimmer Vergangenheit. Siedler, München 2010, ISBN 978-3-88680-949-3.
Der Historiker und der Zeitgenosse. Eine Zwischenbilanz. Siedler, München 2014, ISBN 978-3-8275-0048-9.

Literatur
 Christian Meier: Lehrstuhl Christian Meier. 1.3.1981–31.3.1997. In: Jakob Seibert (Hrsg.): 100 Jahre Alte Geschichte an der Ludwig-Maximilians-Universität München (1901–2001). Duncker und Humblot, Berlin 2002, ISBN 3-428-10875-2, S. 183–195 (Selbstdarstellung der Lehr- und Forschungstätigkeit in München).
 Monika Bernett, Wilfried Nippel, Aloys Winterling (Hrsg.): Christian Meier zur Diskussion. Autorenkolloquium am Zentrum für Interdisziplinäre Forschung in Bielefeld. Steiner, Stuttgart 2008, ISBN 978-3-515-09148-0.

Weblinks

 Deutschlandfunk Kultur Zeitfragen im Gespräch vom 1. Januar 2020: Demokratie in Geschichte, Gegenwart und Zukunft. Der Althistoriker Christian Meier im Gespräch mit Winfried Sträter

Anmerkungen

Althistoriker
Publizist
Sprachkritik
Politische Theorie und Ideengeschichte
Hochschullehrer (Universität Basel)
Hochschullehrer (Ruhr-Universität Bochum)
Hochschullehrer (Albert-Ludwigs-Universität Freiburg)
Hochschullehrer (Universität zu Köln)
Hochschullehrer (Ludwig-Maximilians-Universität München)
Ehrendoktor der Universität Salzburg
Ehrendoktor der Universität Bern
Mitglied des Deutschen Archäologischen Instituts
Mitglied der Norwegischen Akademie der Wissenschaften
Mitglied der Berlin-Brandenburgischen Akademie der Wissenschaften
Mitglied der Deutschen Akademie für Sprache und Dichtung
Mitglied der Academia Europaea
Träger des österreichischen Ehrenzeichens für Wissenschaft und Kunst
Träger des Bayerischen Maximiliansordens für Wissenschaft und Kunst
Deutscher
Geboren 1929
Mann

redpajama_set_name: RedPajamaWikipedia


text:
Home / Insights / CFPB RESPA Enforcement: Alive ...
Andrew T. Hernacki
CFPB RESPA Enforcement: Alive and Kick(back)-ing
On January 31, 2017, the CFPB ordered mortgage lender Prospect Mortgage, LLC to pay a $3.5 million civil penalty for violating the anti-kickback provisions of the Real Estate Settlement Procedures Act (RESPA). According to the consent order, Prospect paid illegal kickbacks to two real estate brokers and a mortgage servicer in exchange for mortgage referrals, in violation of RESPA Section 8(a). The consent order also alleged that the mortgage servicer improperly obtained consumer reports in the form of "trigger leads" in violation of the Fair Credit Reporting Act (FCRA). The Bureau concurrently ordered several brokers and a servicer who accepted the kickbacks from Prospect to pay a combined $495,000 in consumer relief, redress, and penalties.
The Bureau's enforcement action against Prospect and the brokers and servicer who referred business to them is the latest action in CFPB anti-kickback enforcement under RESPA, and is the first RESPA action focused on Marketing Services Agreements (MSAs) since October 2015. Then, the CFPB issued Compliance Bulletin 2015-05, which highlighted the "substantial risks" associated with MSAs and the Bureau's view that many MSAs are designed to evade RESPA's prohibition on the payment and acceptance of kickbacks and referral fees.
Here, the Bureau once again focused on the MSAs between the Sherman Oaks, California-based Prospect Mortgage and its brokers and servicer, contending that Prospect used these and other, similar agreements to funnel illegal kickbacks:
Marketing Services Agreements. Under the MSAs, Prospect allegedly paid its brokers not for marketing services, but rather for volume-based referrals. Prospect allegedly monitored the "capture rate" of its brokers, meaning the percentage of an individual broker's total loan volume referred to Prospect, and adjusted that broker's compensation under the MSA accordingly. Prospect also purportedly held monthly meetings with its MSA-brokers to review their capture rates and discuss "missed opportunities" to refer business to Prospect.
Lead Agreements. Similarly, the Bureau alleged that Prospect entered into exclusive "lead agreements" to pay brokers for referrals. Under these agreements, brokers were prohibited from sending prospective mortgage buyers' (leads) contact information to any of Prospect's competitors, and in turn Prospect paid the brokers on a per-lead basis. This exclusivity provision, however, meant that brokers were strongly incentivized to actively refer prospective buyers to Prospect's loan officers. In addition, Prospect is alleged to have encouraged its brokers to pay the brokers' agents amounts commensurate with the volume of referrals sent to Prospect each month.
Desk Licensing Agreements. Furthermore, the Bureau alleged that Prospect entered into "desk licensing" agreements with various brokers. According to the Bureau, these agreements were not just payments for office space but were actually used to pay brokers to endorse Prospect's services and promote Prospect as a "preferred lender." The key issue, however, was that Prospect analyzed the value of the desk licensing agreements in proportion to the number of referrals, rather than tying the cost of the office space to the fair market value of the particular geographic area.
Pre-approval Requirements. The Bureau also alleged RESPA violations stemming from Prospect's alleged practice of contractually requiring (or otherwise encouraging) brokers to insert a pre-approval requirement into the agent-only remarks section of the local multiple listing service. These instructions required a home buyer's listing agent to inform the buyers that they needed to obtain a mortgage pre-approval specifically with a Prospect loan officer before submitting a purchase offer on the property. In turn, the Bureau alleged that this gave Prospect the "inside track" on the mortgage business because much of the pre-approval information carried over into the full mortgage application.
Fee and Credit Pressure on Consumers. Prospect also allegedly pressured the brokers it worked with to "squeeze" consumers into using Prospect over other lenders. In one example, a broker's agent responded to this pressure by conditioning a discount on the sales price (a "seller's credit") on using Prospect for the mortgage. Other agents imposed a daily penalty on buyers for each day beyond the closing date that the loan did not close, but would waive the fee if the buyer used Prospect.
Unlawful Use of Trigger Leads in Violation of FCRA and RESPA. Prospect allegedly entered into agreements with a mortgage servicer where the servicer obtained "trigger leads" from a credit bureau to identify existing customers seeking to refinance. According to the consent order, the servicer obtained consumer reports but did not make firm offers of credit, in violation of FCRA. Rather, it referred those leads exclusively to Prospect in exchange for the servicing rights for new loans and a split of the proceeds when loans were later sold to secondary purchasers.
This recent action shows that the Bureau continues to be interested in aggressively enforcing RESPA's anti-kickback provisions and takes action not just against non-compliant lenders, but also against the brokers and servicers with whom they work. Further, the Bureau is reinforcing its all-inclusive approach with its statutes, often using its "unfair, deceptive, or abusive acts or practices" authority (or, as here, the FCRA) coupled with a RESPA violation. This action likely serves as a reminder to all people who touch mortgage settlement services to revisit their compliance policies and relationships within the mortgage pipeline. For questions regarding the impact of this action, please contact Venable's Consumer Financial Services Practice Group.
Banking and Financial Services Regulatory
Consumer Financial Services
CFPB Issues FAQs and Softens Tone on Mortgage Marketing Services Agreements
CFPB RESPA enforcement, proposed delay of prepaid rule effective date, and more in this issue of Consumer Financial Services Practice Digest
Inside the CFPB Enforcement Policies and Procedures Manual, movement toward a debt collection proposal, and more in this issue of CFPB Digest
Adjusting Your Operations and Compliance for UDAAP Risks
February 07, 2023 26th Annual Receivables Management Association International (RMAi) Conference
Digital Dark Patterns & Debt Collection
February 08, 2023 Receivables Management Association International
Overview of the Payments System
February 17, 2023 ETA University (ETAU) on the Hill