text:
Q: Excel showing empty cells when importing file created with csv module I have a .csv with rows that are something like:
"Review",Clean Review
"The hotel _was, re3ally good",the hotel was really good

but when I was open with excel 2013 the cells with the double quotes are showing empty on the sheet, and I can only see the text in the formula bar. Can anyone tell me why this is happening?
excel sheet kinda looks like:
|          |Clean Review|
|          |the hotel wa|

I have opened the .csv in notepad and there doesn't seem to be hidden characters causing this behavior
I used Python's csv module to create the .csv
it especially happens if you drag and expand the first column

A: Try
="Review",Clean Review
="The hotel _was, re3ally good",the hotel was really good

This forces the value to be text and not something else.
Haven't tried but theoretically should work.
Please check this and more solutions here

A: The problem is in your CSV file.  There is a CR after the w in Review.  And Review is actually on the 2nd line of A1 (seen if you increase the height of row 1).
Here is an analysis of the characters at the beginning of your actual CSV with the character and the ASCII code.  Note the 10 after the w
"              34
R              82
e              101
v              118
i              105
e              101
w              119
               10
"              34


By Adelle M. Banks/Religion News Service | 4/12/2019, 6 a.m.
About 100 investigators are working to find out who is behind a series of church fires that have struck rural areas of Louisiana in recent weeks.
The fires, which have prompted calls for prayer by the state's governor, destroyed three predominantly African-American churches in St. Landry Parish, around Opelousas, in late March and early April and caused minor damage to a mostly white Pentecostal church in Vivian, about 200 miles north in Caddo Parish.
Ashley Rodriguez, a spokeswoman for the state fire marshal, said investigators have not determined if the three fires at historic Baptist churches in Opelousas and Port Barre are related or if any of them are tied to one at the church in Vivian. But she said the number of investigators is unusually high. It includes 40 federal staffers, including agents from the FBI and the federal Bureau of Alcohol, Tobacco, Firearms and Explosives; 40 state agency workers; and 20 people from local agencies.
Louisiana Gov. John Bel Edwards stayed away from pinpointing a cause, focusing instead on church members' losses.
Rep. Higgins, who is known for previously recording Crime Stoppers announcements for a local TV station, predicted whoever is responsible for the three fires in the parish will be apprehended.


Jeremy Balfour MSP congratulated the Bonnyrigg winners of the top title at Crufts in a motion raised in the Scottish Parliament last week.
Two-and-a-half-year-old whippet Tease and owner Yvette Short took home the Best in Show prize at Crufts 2018 earlier this month.
The Conservative Lothian list MSP said: "I was delighted to hear that the Crufts 2018 trophy is coming to Scotland and congratulate Tease on her prize-winning performance, beating 21,000 dogs.


A beautiful gift pack of three triple milled SAX Luxury Peony Soap Bars which are moisturising and nourishing for your skin.This set contains: 3 x Luxury Soap Bars.
SAX soy candle is fragranced with the exotic and beautiful scent of frangipani. The candle is made of a unique blend of high-quality waxes with a lead-free wick and a remarkable 40-hour burn time.
Beautifully perfumed sachet to place amongst clothes, especially lingerie for a sweet lingering fragrance. Also great in wardrobes or linen cupboard. Ingredients Vermiculite, rose fragrance.
LIMITED EDITION - Gift box makes the perfect gift for any occasion.Contains Lemon & Lavender Glycerine Soap 100gms and a Lavender & Orange Lip Balm (with sunscreen).
This is a wonderful way to create a fresh, inviting ambience in your home. Simply spray around the home to refresh any room with a lingering Lavender fragrance. Ingredients Deionised water, poly sorbate 20, alcohol, phenoxyethanol, lavender oil.
A classic luxury item. Romantic, tender and luxuriously perfumed with Bulgarian Rose Oil. Dust over body after bathing to help seal in moisture and leave skin soft and smooth. Ingredients Sterislised talc, zinc stearate, magnesium carbonate, perfume.
A classic luxury item. Refreshing, naturally perfumed powder. Dust over body after bathing to help seal in moisture and leave skin soft and smooth. Ingredients Sterised talc, zinc stearate, magnesium carbonate, lavender oil.
A light, easily absorbed body oil enriched with powerful almond oil and essential oils. This rich oil used in the bath or directly onto skin works to protect the skins elasticity. The deeply nourishing formula restores and balances dry and damaged skin. Ingredients Almond oil, rose fragrance, rose oil, sandalwood oil, ylang ylang oil.
Aromatic massage oil for general use. Contains pure essential oils of Lavender, Geranium and lemon in a base of Sweet Almond Oil. Softely conditions the skin while relaxing muscles. Ingredients Almond oil, lavende roil, geranium oil, lemon oil.
Unwrap and indulge with Scullys Luxury Soap. Make your everyday bathing ritual a delight, a creamier more luxuriant lather that leaves the skin soft and smooth. Enriched with nourishing Shea Butter and Vitamin E. Ingredients Sodium palm kernelate, shea butter, fragrance, stearic acid, tetrasodium EDTA, etidronic acid, yellow oxide.
Unwrap and indulge with Scullys Luxury Soap. Make your everyday bathing ritual a delight, a creamier more luxuriant lather that leaves the skin soft and smooth. Enriched with nourishing Shea Butter and Vitamin E. Ingredients Sodium palm kernelate, shea butter, lavender oil, stearic acid, tetrasodium EDTA, etidronic acid, yellow oxide.


SOME WEEKS AGO WHEN THE WEATHER WAS STILL COLD I STOOD IN THE BIG GLASS WINDOW IN CORNMARKET STREET IN OXFORD IN A SHOP OF RIVER ISLAND TOGETHER WITH SOME OTHER FIGURES OF MY KIND IN DIFFERENT MAKEUP.
SHE WAS OBVIOUSLY IN A HURRY.
SHE CAME BACK LATER AND BOUGHT ME DURING A LENGTHY DISCUSSION WITH A REALLY NICE SALES ASSISTANT WHO HELPED HER TO MAKE ME TRANSPORTABLE IN A BIG BOX AND SHE LEARNT THAT I HAD LANDED THERE IN THE WINDOW THRU THE CHARITY WORK OF GREENFINGERS.
ANYWAY I LANDED ON THE BLACK CAN-ON PRINTER next to her writing desk watching her thinking and writing and pondering.
"oh she emailed that she needs a photo because she misses him."
i am a robot thing in accelerated intelligence learning training mode.
i do not have emotions.
but not suffer from them.
i am in her mind.
where the software is stored in this visible hardware.
" RoLord Hic Upp Sir Esquire"
! l e t s d o i t !


text:
AV Daily News
City of Palmdale's Courson Arts Colony Wins SCAG Sustainability Award for Housing Innovation
PALMDALE - The Southern California Association of Governments (SCAG) has announced that the City of Palmdale's Courson Arts Colony won the 2020 Sustainability Award in the Housing Innovation category.
SCAG's Sustainability Awards program annually recognizes excellence in coordinating land use and transportation to improve mobility, livability, prosperity and sustainability. The program received a record number of applications this year.
"We are honored and pleased to be recognized by SCAG with this prestigious award for such a remarkable project," said Palmdale Mayor Steve Hofbauer. "Our team took an area that had previously struggled and transformed it into quality housing and amenities to help those who need it the most."
Over the past twenty years the City redeveloped three City blocks in the downtown area, including construction of a new senior center and 153 units of affordable housing for seniors; acquired and revitalized 291 affordable housing units; replaced the first community swimming pool in Courson Park; and most recently completed the final phase of development known as the Courson Arts Colony. The Colony consists of 159 affordable housing units with the development being focused around the cultivation and promotion of the arts. All apartments are equipped with fully furnished kitchens.
Courson Arts Colony's common areas consist of a fitness room, a dance studio, art gallery, an art walk, flexible art space, digital art room, clubhouse room, homework room, leasing room, and an outdoor amphitheater. It contains 60 permanent supportive housing apartments that will serve medically complex veteran and non-veteran households experiencing homelessness.
"I could not be any prouder of our staff who have collaborated over the years to create a unique experience that provides much needed housing and important resources," said Palmdale City Manager J.J. Murphy. "Along with the affordability aspect of this project, it carved out opportunities for artists and had 40 apartments set aside for veterans. The arts component serves and provides an environment for artists or those who wish to live in an artist-centric environment, as well as serving as a healing element for our veterans living here who may be suffering from Post-Traumatic Stress Disorder (PTSD). Classes and support staff are on hand to work with and encourage tenants to utilize the amenities and explore their artistic spirit."
Due to the cancelation of SCAG's 2020 Regional Conference and General Assembly, the Sustainability Award winners will unfortunately not be honored at a formal ceremony, but the video for Palmdale's award may be viewed here: https://vimeo.com/415272764
About SCAG
Founded in 1965, the Southern California Association of Governments (SCAG) is a Joint Powers Authority under California state law, established as an association of local governments and agencies that voluntarily convene as a forum to address regional issues. Under federal law, SCAG is designated as a Metropolitan Planning Organization (MPO) and under state law as a Regional Transportation Planning Agency and a Council of Governments.
The SCAG region encompasses six counties (Imperial, Los Angeles, Orange, Riverside, San Bernardino and Ventura) and 191 cities in an area covering more than 38,000 square miles. The agency develops long-range regional transportation plans including sustainable communities strategy and growth forecast components, regional transportation improvement programs, regional housing needs allocations and a portion of the South Coast Air Quality management plans.
To learn more about SCAG, please visit www.scag.ca.gov/ .

redpajama_set_name: RedPajamaCommonCrawl


text:
Powerpuff Girls: Defenders of Townsville Review
Jen Schiller
The Powerpuff Girls are back, and they are back in a huge way. After years off the air, these pint-sized superheroes got a major makeover, and now they're returning to screens of all sizes. While the new cartoon is yet to premiere, the Android game, Defenders of Townsville, is an excellent way to spend the days waiting for their updated TV debut.
Defenders of Townsville is designed an awful lot like Metroid, and not just because the heroes are lacking a Y-Chromosome. The game takes advantage of the well-established world in which the Powerpuff Girls live: we know Mojo Jojo is up to no good simply because he's awake and near the girls, who have been stripped of their powers, and need to earn them back by taking out Mojo's evil robots.Our pint-sized heroines have no choice but to fight their way through room after room in search of modules that will help return them to their rightful places of control. When the game begins, only Buttercup is playable. She's bound by gravity and can only throw a little punch in defense. After a while, and some updates to her attack, Buttercup releases Blossom in a boss battle, and then Blossom releases Bubbles in a battle of her own. Each girl has specific powers (Buttercup has a wide range of melee attacks, Blossom can fight with fire, and Bubbles is all about shooting ice.), but there are also power modules that allow all three girls to fly faster, fight harder, or last longer against enemy fire.
Here are some things Defenders of Townsville gets very VERY right: three levels of difficulty, an easy to navigate control system, and interchangeable animation styles. This is a game that both I and my nine year old cousin are equally likely to grab for and play. Cartoon Network planned for the learning curve between the game's two biggest audiences with "Easy" "Normal" and "Hard" levels. The controls system; which consists of a joystick on the left side of the screen and the "fly" and "attack" buttons on the right, streamlines gameplay so that you can move and fight at the same time. Another button above the joystick makes it easy to change your special attack without taking away from your navigation. A fourth button above the attack allows you to speed up flight once you've unlocked that module.
I mentioned interchangeable animation before, and while this is an amazing feature of the game it's almost an Easter egg. I'm not one to mess with the settings of a game—if it's intended to be played a certain way I follow that template until I find a problem with it. However, while reading up on the newest Powerpuff Girls series, I stumbled upon another article about Defenders that was over the moon about the classic animation that's included. I poked around in the settings and voila! The girls were back! I do like the new animation style and I'm really excited to see it at play in the series, but I have to say those smooth edges and bolder colors were like a sight for sore eyes after hours and hours of sugary-sweet pastels and hard-corner geometry.
The only problem I had with Defenders, and a problem that many had with Metroid as well, is the pacing. I found myself going back over rooms I thought I scoured, just because I had a new power, on the off-chance it uncovered a module. Once I got a map with indicators, I spent slightly less time chasing my own tail—but only slightly. This game requires that you retrace your steps, which is fine, but these girls are known for saving the world before bedtime…not the break of dawn.
Once you start getting some powers back, though, the pace starts to grow exponentially, and the problem starts to pale in comparison to all the good stuff. Overall, Defenders of Townsville is a solid Metroid-style game. It's got a sprawling map that's simple to navigate. While I would have liked a more in-depth storyline, a game that's helping re-introduce the show for a new audience in equal measure to its older fans does well to skim the surface with a classic Big Bad.
Gameplay in Defenders of Townsville is smooth and fun; load time between rooms is negligible and the urge to 100% clear a room of enemies is both tempting before it's accomplished, and satisfying after. There is nothing here that doesn't belong, which makes it a clean representation of the Townsville world, and a well-developed addition to the Metroidvania family. I dare you to play it without a smile.
Is it Hardcore?
Definitively.
If the game were a little more fast-paced, with a deeper storyline, I would have no problem rewarding it a five. As it stands, Defenders of Townsville is a rockin' game for the Powerpuff in all of us.
Jen Schiller is a freelance writer with a penchant for nerdy subject matter. Her gaming career began with Aladdin on the Genesis, and she is still hard-pressed to name a platformer she enjoys more. Find her on twitter @jenisaur.
Solarmax 2 Review
John Markley
Wayward Souls Review
Tyler Zelenko
Mario Tantillo
WWE Undefeated Review
James Williamson
Bloodstained: Ritual of the Night Review
Moon Raider Review
MARVEL Realm of Champions Review
Zach Guida


A GtkSpinner widget displays an icon-size spinning animation.
It is often used as an alternative to a ProgressBar for displaying indefinite activity, instead of actual progress.
To start the animation, use start, to stop it use stop.
GtkSpinner has a single CSS node with the name spinner. When the animation is active, the GtkSpinner:checked pseudoclass is added to this node.
Returns a new spinner widget.
Starts the animation of the spinner.
Stops the animation of the spinner.


text:
\section{Introduction}
Stochastic gradient descent (SGD) methods are workhorse methods for solving large-scale optimization problems arising in machine learning \citep{bottou2010large,bottou2012stochastic,ruder2016overview}. SGD has good scalability properties to large datasets and to high dimensions and often leads to solutions that can generalize well, i.e. perform well on unseen data. These favorable properties are among the key reasons why SGD is a default method both in academia and in industry for training predictive models in deep learning or more broadly in supervised learning tasks \citep{lecun2015deep,goodfellow2016deep,le2011optimization}.
\looseness=-1

SGD iterates move along the stochastic (noisy) estimates of the gradient of the underlying optimization objective. Recent studies on SGD methods in the last few years unveiled a mysterious ``heavy-tailed behavior" that can be observed in different ways during stochastic optimization.
More specifically, it has been numerically observed that the gradient noise becomes often heavy-tailed with a non-Gaussian structure over iterations in deep learning practice (even if at the beginning of the SGD iterations the gradient noise may have a Gaussian-like structure) \citep{csimcsekli2019heavy,pmlr-v97-simsekli19a,ht_sgd_quad}.
\citet{martin2019traditional, martin2020heavy} also showed that the network weights and the eigenvalues of weight matrices show a heavy-tailed behavior for well-trained networks. They proposed to fit a power law distribution to the empirical spectral density of the weight matrices and illustrated that heavier-tailed weight matrices indicate better generalization.
It has been also observed that the level of heaviness of the tails and non-Gaussianity depends on the SGD hyperparameters and the geometry of the landscape of the objective, smaller batch sizes and larger stepsize being associated to heavier tails \citep{panigrahi2019non,ht_sgd_quad}. Formalizing these empirical observations, recent theoretical studies \citep{ht_sgd_quad, hodgkinson2020multiplicative} showed that heavy tails arise due to the ``multiplicative noise" in stochastic optimization, and can arise even under surprisingly simple settings such as linear regression with Gaussian data. In particular, G\"urb\"uzbalaban \emph{et al.} showed that for least square problems subject to Gaussian data, the tails are monotonic with respect to the stepsize and the batch \citep{ht_sgd_quad} and there exists a range of stepsizes for which there exists a stationary distribution with an infinite variance. It has also been found that the heaviness of the tails are positively correlated with the generalization performance on a number of datasets and architectures \citep{simsekli2020hausdorff,barsbey2021heavy}.

Aforementioned results from the literature about the heavy tails apply to conventional stochastic optimization problems (such as risk minimization), where all the data is assumed to be available at a centralized particular location. On the other hand, modern datasets are often too large to be handled by a single machine (or a processor) and are commonly collected, stored, and processed in a distributed manner over a network of computing (nodes) agents. This fact, together with increasing data privacy requirements, necessitated the development of communication-efficient
algorithms for solving \emph{decentralized stochastic optimization} problems of the form\looseness=-1
\begin{equation}
\min\nolimits_{x \in \mathbb{R}^d} f(x):= \sum\nolimits_{i=1}^N f_i(x), \quad  f_i(x):=\mathbb{E}_{z_i\sim\mathcal{D}_i}[\ell(x, z_i)],\label{pbm-pop-min}
\end{equation}
where $\ell(x,z_{i})$
represents the instantaneous \emph{loss} at node $i$ based on the predictor $x$ and the data point $z_i$, where there are $N$ computation nodes lying on a connected undirected graph $G=(\mathcal{N},\mathcal{E})$. Here $\mathcal{N} = \{1,2,\dots,N\}$ is the set of (vertices) nodes and $\mathcal{E}\subseteq \mathcal{N}\times \mathcal{N}$ is the set of edges that define the connectivity patterns between the nodes. The objective $f_i$ is only available at the node $i$, and the aim is to
train models locally at each agent where only local parameters vectors are shared among the neighbors.
The problem \eqref{pbm-pop-min} arises in a number of key settings including but not limited to decentralized deep learning \citep{dean2012large,ying2021exponential} and federated learning \citep{li2020review}.

Decentralized versions of SGD that are based on a weighted averaging of local parameter vectors among neighbors are popular for solving \eqref{pbm-pop-min} where the weights are specified through an $N\times N$ stochastic matrix called the \emph{communication matrix } or the \emph{mixing matrix} \citep{robust-network-agd,gurbuzbalaban2021decentralized}. There are two popular decentralized versions of SGD, first version averages both local parameter vectors and gradients \citep{pu2020asymptotic,xin2020improved} whereas the second version averages only local parameter vectors \citep{robust-network-agd,gurbuzbalaban2021decentralized,ram2009asynchronous, chen2012diffusion,lian2017can}. Both versions admit similar updates, they attain similar convergence rates at least for strongly convex stochastic optimization problems \citep{robust-network-agd,pu2020asymptotic}.
In this paper, we will focus on the latter version which has a long history at least going back to decentralized gradient methods studied in \citet{nedic2009distributed}. \looseness=-

A natural question in this context is whether the ``heavy-tail phenomenon" arises in the decentralized SGD setting as well, and what would be the effect of decentralization on the tails; i.e. how would the heaviness of tails differ from the default centralized SGD (C-SGD) setting.
To our knowledge, none of the existing theoretical works about heavy tails apply to the decentralized stochastic optimization, which will be the focus of this paper. Our contributions are summarized as follows:
\begin{itemize}[noitemsep,topsep=0pt,leftmargin=*,align=left]
    \item We show in Proposition~\ref{prop:ht_noncvx} that if the local objectives $f_i$ are twice continuously differentiable and strongly convex outside a compact region, then DE-SGD iterates are heavy-tailed at stationarity, provided that the stepsize is small enough.
    This result applies to a general class of objectives that can be non-convex on a compact set. Its proof is based on (i) relating the DE-SGD iterations in dimension $d$ to C-SGD iterations in higher dimensions (in dimension $Nd$) on a ``modified" objective where $N$ is the number of agents, (ii) showing by a careful analysis that the ``modified" objective inherits strong convexity properties outside a compact set in higher dimensions as well provided that the stepsize is small enough, (iii) building on existing results for C-SGD from \citet{hodgkinson2020multiplicative}. \looseness=-
    \item To get more precise characterizations of the tail behavior, similar to \citet{ht_sgd_quad}, we consider
    least square problems (where the losses $f_i$ are
quadratics) when the underlying data has a continuous distribution with finite moments. Our result (Theorem~\ref{thm:main:dsg}) shows that the iterates converge to a heavy-tailed distribution whose tail probabilities decay polynomially at a specific rate $\alpha$.
The decay rate $\alpha$
is hard to compute exactly in general; however, we discuss how it can be estimated as a unique positive solution $\hat{\alpha}$ to a non-linear equation $\hat h(s) = 1$ where $\hat h$ is a function of the stepsize, batch-size, the curvature of the objective and the mixing matrix. Furthermore, building on the connections between C-SGD and DE-SGD summarized in the previous bullet point, we show in Theorem~\ref{thm:mono} that the tail-index $\hat{\alpha}$ is strictly increasing in batch sizes $b_i$ at every node $i$ and strictly decreasing in stepsize $\eta$ provided that $\hat\alpha\geq 1$. These results show that the effect of hyperparameters in the decentralized setting resembles to the centralized setting previously studied in \citet{ht_sgd_quad}. \looseness=-1
\item
To complement Theorem~\ref{thm:main:dsg}, that is of asymptotic nature,
we provide non-asymptotic moment
bounds for $k$-th iterate and also characterize the non-asymptotic convergence speed
from the $k$-th iterate to the stationary limit
in Wasserstein metric
in the Appendix.
\item To understand the effect of decentralization and network averaging, we compare the tail-index of DE-SGD to that of C-SGD and disconnected SGD (Dis-SGD); the latter algorithm runs independent copies of SGD at each node without connecting them through a network while distributing the data points evenly. Here, Dis-SGD behaves like an intermediate point between C-SGD and DE-SGD and serves as a benchmark. We first infer from existing results about C-SGD that Dis-SGD has heavier tails compared to C-SGD (Proposition~\ref{prop:disconnected:centralized}). Then, under Gaussian input data assumption, we show in Theorem \ref{thm:first:order:alpha} that if the network size is large enough or if the stepsize is large enough, DE-SGD has heavier tails compared to Dis-SGD under a mild technical assumption which ensures that the stationary distribution exists. On the other hand, when the stepsize is small enough or if the network is small enough, we show that DE-SGD has lighter tails compared to Dis-SGD. These results allow us to compare DE-SGD with C-SGD showing that DE-SGD has heavier tails than C-SGD for some range of parameters (Corollary~\ref{cor:DSGD:CSGD}). In fact, our theory uncovers two regimes of parameters (stepsize and network size), where addition of the network links can make the tails lighter or heavier (compared to the Dis-SGD setting). This phenomenon is demonstrative of rich interplays between the heavy tails and the network structure, and we believe our work is the first step towards understanding this interaction. In the main text, due to space limitations we consider the simpler (and more intuitive) $d=1$ case, results about general dimension $d$ can be found in the Appendix.
\item Adapting the results of \citet{gao2015stable} to our setting, we provide a generalized central limit theorem for the averaged iterates in the Appendix which show that they follow an $\alpha$-stable distribution in the limit. This result allows us to estimate the tails more accurately in our experiments based on recent advanced estimators available for $\alpha$-stable distributions. Finally, we provide numerical experiments conducted on both synthetic data and neural
networks to empirically support and illustrate our theoretical results. For both fully-connected networks and modern networks with convolutional layers such as ResNet-20, we provide experiments for DE-SGD that illustrate the heavy-tailed behavior in practice.

\end{itemize}

\section{Decentralized SGD and Preliminaries}
\label{sec:bg}
\textbf{Decentralized SGD (DE-SGD).}
A standard method for decentralized learning is the decentralized stochastic gradient (DE-SGD) method
\citep{robust-network-agd, Yuan16, ram2009asynchronous, lian2017can}. At iteration $k$, DE-SGD updates the local variable $x_{i}^{(k)}$ at node $i$ with the following recursion
\begin{equation}\label{eqn:dsg_update}
x_{i}^{(k+1)}=\sum\nolimits_{\ell\in\Omega_{i}}W_{i\ell}x_{\ell}^{(k)}-\eta\tilde{\nabla} f_{i}(x_{i}^{(k)}),
\end{equation}
where $\eta>0$ is the stepsize, $W \in \mathbb{R}^{N\times N}$ is a symmetric double stochastic matrix also known as the communication (mixing) matrix, with $W_{ij}=W_{ji}>0$ if $j\in\Omega_{i}$,
and $W_{ij}=W_{ji}=0$ if $j\not\in \Omega_{i}$ and $i\neq j$,
and finally $W_{ii}=1-\sum_{j\neq i}W_{ij}>0$
for every $1\leq i\leq N$.\footnote{We refer the reader to the Appendix for more about the properties of the matrix $W$.} $\Omega_i$ is the set of neighbors of node $i$ on the network, i.e. $\Omega_i = \{ j : (i,j) \in \mathcal{E}\}$. Moreover, in \eqref{eqn:dsg_update}, $\tilde{\nabla} f_{i}(x)$ is an estimate of the gradient of the loss $f_i(x)$ at node $i$ based on a batch size $b_i$, i.e. it is based on $b_i$ random draws from data satisfying,
\begin{equation} \tilde{\nabla} f_{i}( x_i^{(k)})
 := (1/b_i) \sum\nolimits_{j=1}^{b_i} \nabla\ell (x_i^{(k)}, z_{i,j}^{(k)}),
 \label{eq-stoc-grad}
\end{equation}
where $z_{i,j}^{(k)}$ are (fresh) i.i.d. draws from data at step $k$. \looseness=-1

\textbf{Heavy-tailed distributions with a power-law decay.}
A real-valued random variable $X$ is said to be \emph{heavy-tailed}
if the right tail or the left tail of the distribution  decays slower than any exponential distribution.
We say $X$ has heavy (right) tail
if $\lim_{x\rightarrow\infty}\mathbb{P}(X\geq x)e^{cx}=\infty$
for any $c>0$,
and a real-valued random variable $X$ has heavy (left) tail
if $\lim_{x\rightarrow\infty}\mathbb{P}(X\leq-x)e^{c|x|}=\infty$
for any $c>0$; see e.g. \citep{Schmidli2009}. Similarly, an $\mathbb{R}^{d}$-valued random vector $X$
has heavy tail if $u^{T}X$ has heavy right tail
for some vector $u\in\mathbb{S}^{d-1}$,
where $\mathbb{S}^{d-1}:=\{u\in\mathbb{R}^{d}:\Vert u\Vert=1\}$ is the unit sphere.

Heavy tail distributions include $\alpha$-stable distributions,
Pareto distribution, log-normal distribution and the Weilbull distribution. One important class of the heavy-tailed distributions
is the distributions with \emph{power-law} decay,
which is the focus of our paper. That is,
$\mathbb{P}(X\geq x)\sim c_{0}x^{-\alpha}$
as $x\rightarrow\infty$ for some $c_{0}>0$ and $\alpha>0$,
where $\alpha>0$ is known as the \emph{tail-index},
which determines the tail thickness of the distribution.
Similarly, we say that the random vector $X$ has power-law decay
with tail-index $\alpha$ if for some $u\in\mathbb{S}^{d-1}$,
we have $\mathbb{P}(u^{T}X\geq x)\sim c_{0}x^{-\alpha}$,
for some $c_{0},\alpha>0$.
For a reference on heavy-tailed and power-law distributions, we refer to \citet{Foss2013}

\textbf{Viewing DE-SGD as centralized SGD in higher dimensions.}
As in \citet{robust-network-agd,Yuan16}, we can express the DE-SGD iterations as
$x^{(k+1)}=\mathcal{W}x^{(k)}-\eta \tilde{\nabla} F\left(x^{(k)}\right)$,
where $\mathcal{W} := W \otimes I_d$, $F:\mathbb{R}^{Nd}\rightarrow\mathbb{R}$ defined as
$F(x):=F(x_{1},\ldots,x_{N})
=\sum\nolimits_{i=1}^{N}f_{i}(x_{i})$,
with $x^{(k)}:=[(x_{1}^{(k)})^{T},(x_{2}^{(k)})^{T},\ldots,(x_{N}^{(k)})^{T}]^{T}\in\mathbb{R}^{Nd}$
and
\begin{equation}
\tilde{\nabla}F(x^{(k)}):=[(\tilde{\nabla} f_1 (x_1^{(k)}))^{T},\ldots,(\tilde{\nabla} f_N (x_N^{(k)}))^{T}]^{T}.
\label{def-stoc-grad}
\end{equation}
We can alternatively view DE-SGD as C-SGD iterations
\begin{equation}
x^{(k+1)} = x^{(k)} - \eta \tilde \nabla F_{\mathcal{W}}(x^{(k)})
\label{eq-dsgd-vs-sgd}
\end{equation}
on a modified objective $F_{\mathcal{W}}$ defined as
\begin{equation}
F_{\mathcal{W}}(x) := F(x) +  (1/2)\eta^{-1}x^T (I_{Nd}-\mathcal{W}) x, \label{def-FW}
\end{equation}
with the convention that
$\tilde \nabla F_{\mathcal{W}}(x) = \tilde \nabla F(x) + \frac{1}{\eta} (I_{Nd}-\mathcal{W}) x$
(see e.g. \citep{robust-network-agd}). Similar to \eqref{def-stoc-grad}, we can define the stochastic Hessian as $\tilde{\nabla}^2 f_{i}( x_i^{(k)})
 := \frac{1}{b_i} \sum_{j=1}^{b_i} \nabla^2 \ell(x_i^{(k)}, z_{i,j}^{(k)})$ with
\begin{equation}
\tilde{\nabla}^2 F(x^{(k)}):=[(\tilde{\nabla}^2 f_1 (x_1^{(k)}))^{T},\ldots,(\tilde{\nabla}^2 f_N (x_N^{(k)}))^{T}]^{T},
\label{def-stoc-hessian}
\end{equation}
when $f_i$'s are twice differentiable for every $i=1,2,\dots,N$.

\section{Decentralized SGD and Heavy Tails}
\label{sec-least-squares}

\textbf{3.1 General smooth loss. }
We begin by the following result, which is an extension of \cite[Theorem~1]{hodgkinson2020multiplicative}, from centralized to the decentralized setting. It shows that if the objectives $f_i$ are strongly convex outside a compact set for every $i$ and if the stepsize is small enough, then the iterates will admit a heavy-tailed distribution. The proof is based on showing that under these assumptions, the function $F_{\mathcal{W}}(x)$ defined in \eqref{def-FW} is strongly convex outside a compact set for $\eta$ small and then using the identity \eqref{eq-dsgd-vs-sgd} that relates DE-SGD to C-SGD. Under our assumptions, the function $F(x)$ is not necessarily strongly convex outside a compact set (because when $x$ is large some coordinates $x_i$ can potentially be arbitrarily small); and the quadratic term in \eqref{def-FW} can vanish on a subspace. However, by a careful analysis, we show that outside a large enough compact set, when $F(x)$ does not have strong convexity then the quadratic term does have it so that the sum $F_{\mathcal{W}}(x)$ defined in \eqref{def-FW} is strongly convex as desired.

\begin{proposition}
\label{prop:ht_noncvx}
Assume that $f_i(x)$ is twice continuously differentiable on $\mathbb{R}^d$ and each $f_i$ is strongly convex outside a compact region in $\mathbb{R}^d$ for every $i=1,2,\dots, N$. Define
$\mathfrak{L}_{\text{up}}:= \sup_{x\in\mathbb{R}^{Nd}} \|\mathcal{W} - \eta \tilde{\nabla}^2 F(x) \|$ and
$\mathfrak{L}_{\text{low}}:=  \liminf_{  \|x\| \to \infty } \sigma_{\mathrm{min}} ( \mathcal{W} - \eta \tilde \nabla^2 F(x) )$,
where $\tilde \nabla^2 F(x)$ is as in \eqref{def-stoc-hessian}, $\sigma_{\mathrm{min}} (A)$ denotes the smallest the singular value of a matrix $A$.
Further assume that the following holds:
$\mathbb{E} \left[\log \mathfrak{L}_{\text{up}}\right] < 0, \quad \mathbb{E} \left[\mathfrak{L}_{\text{up}}\right] < \infty, \quad \mathbb{P}(\mathfrak{L}_{\text{low}} > 1) > 0$.
Then, for $\eta$ small enough, there exists $\alpha,\beta >0$ such that $\mathbb{E} \left[\mathfrak{L}_{low}^\alpha\right] = \mathbb{E}\left[\mathfrak{L}_{up}^\beta\right] = 1$ and for every $\varepsilon >0$,
$\limsup_{t \to \infty} t^{\alpha+\varepsilon} \mathbb{P}\left(\|x^{(\infty)}\|>t\right) >0$
and
$\limsup_{t \to \infty} t^{\beta-\varepsilon} \mathbb{P}\left(\|x^{(\infty)}\|>t\right) < \infty$.
\end{proposition}
Proposition~\ref{prop:ht_noncvx} shows that the norm of the iterates is heavy-tailed with a tail-index lying between $\beta$ and $\alpha$. In the following section, we show that for least square problems (where the losses $f_i$ are quadratics), we can get stronger results
which characterize the tail-index
in a more precise manner.

\textbf{3.2 Quadratic loss. }
In the specific case of decentralized least squares, the loss function in \eqref{pbm-pop-min} is a quadratic of the form $ \ell(x,z_i) =\frac{1}{2}(a_i^T x - y_i)^2 \quad \mbox{for every node $i$},$
where $z_i = (a_i,y_i)$ is the local data at agent $i$ with $a_i$ representing the input feature vector and $y_i$ being the output. Recalling that each node $i$ has access to $b_i$ samples from data $\{z_{i,j}^{(k)} =(a_{i,j}^{(k)},y_{i,j}^{(k)})\}_{j=1}^{n_i}$ at every iteration $k$ to form a stochastic gradient estimate, \eqref{eq-stoc-grad} becomes
$\tilde{\nabla} f_{i}(x_i^{(k)})
 := \frac{1}{b_i} \sum_{j=1}^{b_i}[ a_{i,j}^{(k)}(a_{i,j}^{(k)})^T x_i^{(k)} - y_{i,j}^{(k)} a_{i,j}^{(k)}]$.
DE-SGD iterations then become
\begin{equation}\label{eq-iter-dgd}
x^{(k+1)}= M^{(k+1)} x^{(k)} + q^{(k+1)},
\quad
\text{where}
\quad
M^{(k+1)} := \mathcal{W} - \eta H^{(k+1)},
\end{equation}
with
$H^{(k+1)} := \mbox{blkdiag} ( \{ H_{i}^{(k+1)} \}_{i=1}^N )$,
$q^{(k+1)} := [(q_{1}^{(k+1)})^{T},(q_{2}^{(k+1)})^{T},\ldots,(q_{N}^{(k+1)})^{T}]^{T}$ where for $i=1,2,\dots,N\,$
\begin{align}
H_{i}^{(k+1)} := (1/b_i)\sum\nolimits_{j = 1}^{b_i} a_{i,j}^{(k)} (a_{i,j}^{(k)})^T,
\quad
q_i^{(k+1)} := (\eta/b_i) \sum\nolimits_{j=1}^{b_i} a_{i,j}^{(k)} y_{i,j}^{(k)}\,,\label{def-Hk}
\end{align}
where $a_{i,j}^{(k)}$ and $y_{i,j}^{(k)}$ are i.i.d. over $k$ random draws from the data
with the same distribution as $a_{i,j}, y_{i,j}$ that satisfy the following assumptions:

\begin{itemize}
    \item [\textbf{(A1)}] For every $i$, $a_{i,j}$ are i.i.d. over $j$ following a continuous distribution supported
on $\mathbb{R}^d$ with all the moments finite.

    \item [\textbf{(A2)}] For every $i=1,2,\dots,N$, $y_{i,j}$ are i.i.d. over $j$ with a continuous density whose support is $\mathbb{R}$ with all the moments finite.
\end{itemize}

We assume $\textbf{(A1)}$ and $\textbf{(A2)}$ throughout the paper,
and they are satisfied in a large variety of cases, for instance when $a_{i,j}$ and $y_{i,j}$ are Gaussian distributed. We recall the concatenated iterates
$x^{(k+1)}=M^{(k+1)}x^{(k)}+q^{(k+1)}$,
where $M^{(k+1)},q^{(k+1)}$ are defined by \eqref{eq-iter-dgd}, \eqref{def-Hk}.
Let us introduce
\begin{equation}
h(s) := \lim\nolimits_{k\to\infty}(\mathbb{E}\| M^{(k)} M^{(k-1)}\dots M^{(1)}\|^s)^{1/k}\,,
\label{def-hs}
\end{equation}
which arises in stochastic matrix recursions (see e.g. \citep{buraczewski2014multidimensional}) where $\|\cdot\|$ denotes the matrix 2-norm (i.e. largest singular value of a matrix).
Since $\mathbb{E}\left\|M^{(k)}\right\|^s < \infty$ for all $k$ and $s>0$, we have $h(s) < \infty$.
Let us also define $\Pi^{(k)} := M^{(k)} M^{(k-1)}\dots M^{(1)}$ and
\begin{equation}
\rho := \lim\nolimits_{k\to\infty} (2k)^{-1} \log(\mbox{largest eigenvalue of } (\Pi^{(k)})^T \Pi^{(k)})\,.
\label{def-rho}
\end{equation}
The latter quantity is
called the top Lyapunov exponent of the stochastic recursion (\ref{eq-iter-dgd}).

In the following, by following similar arguments as in \citet{ht_sgd_quad},
we can show that the limit density
has a polynomial tail
with a tail-index given precisely by $\alpha$, the unique critical value
such that $h(\alpha)=1$.
The result builds on adapting the techniques developed in stochastic matrix recursions \citep{alsmeyer2012tail,buraczewski2016stochastic} to our setting. Our result shows that even in the simplest setting when the input data is i.i.d. without any heavy tail, DE-SGD iterates can lead to a heavy-tailed stationary distribution with an infinite variance.

\begin{theorem}\label{thm:main:dsg}
Suppose Assumptions \textbf{(A1)}-\textbf{(A2)} hold.
Consider the DE-SGD iterations \eqref{eq-iter-dgd}.
If $\rho<0$ and there exists a unique positive $\alpha$ such that $h(\alpha)=1$, then
\eqref{eq-iter-dgd} admits a unique stationary solution $x^{(\infty)}$
and the DE-SGD iterations converge to $x^{(\infty)}$ in distribution,
where the distribution of $x^{(\infty)}$ satisfies
$\lim\nolimits_{t\to\infty} t^\alpha \mathbb{P}\left(u^T x^{(\infty)} > t \right)= g_\alpha(u)$, for any $u\in\mathbb{S}^{Nd-1}$,
for some positive and continuous function $g_\alpha$ on
$\mathbb{S}^{Nd-1}$.
\end{theorem}
\textbf{Generalized CLT (GCLT).} Following Theorem~\ref{thm:main:dsg}, one can derive a generalized central limit theorem (GCLT) result
that shows that when properly scaled, the sum of the iterates $S_{K}:=\sum_{k=1}^{K} x^{(k)}$ converges in law to a stable distribution.
We refer to
the Appendix
for the details.

Theorem \ref{thm:main:dsg} provides a formula for the tail-index $\alpha$. However, since $\rho$ and $h(s)$ do not have simple closed-form formulas, it is hard to write an explicit formula for $\alpha$ in Theorem~\ref{thm:main:dsg}. In fact, computing $\rho$ is hard in general \citep{tsitsiklis1997lyapunov}. That being said,
using the sub-multiplicativity of the norm of matrix products appearing in \eqref{def-hs} and \eqref{def-rho}, they can be upper bounded as follows:\looseness=-1
\begin{align}
\rho\leq\hat{\rho}:=\mathbb{E}\log\left\Vert\mathcal{W}-\eta H\right\Vert,
\qquad
h(s) \leq \hat{h}(s):=\mathbb{E}\left[\left\Vert\mathcal{W}-\eta H\right\Vert^{s}\right],
    \label{ineq-h-s}
\end{align}
where $H$ is a matrix that has the same distribution as $H^{(k+1)}$ (which does not depend on $k$).

\textbf{Lower bounds on the tail-index $\alpha$.} If $\hat{\alpha}$ is such that $\hat{h}(\hat\alpha)=1$, then by (\ref{ineq-h-s}), $\hat{\alpha}$ is a lower bound on the tail-index $\alpha$ that satisfies $h(\alpha)=1$ where $h$ is defined as in (\ref{def-hs}). In other words, we have $\hat{\alpha}\leq \alpha$ and therefore $\hat{\alpha}$ serves as a lower bound on the tail-index.

\textbf{Checking the conditions in Theorem~\ref{thm:main:dsg}.} We remark that $\hat{\rho}$ and $\hat{h}(s)$ can help us check the conditions in Theorem~\ref{thm:main:dsg}.
Since $\rho\leq\hat{\rho}$, we have $\rho<0$ when
$\hat{\rho}<0$. Moreover, $h(0)=\hat{h}(0)=1$, and one can check that $h(s)$ is convex in $s$. When $\hat{\rho}<0$, $\hat{h}'(0)=\hat{\rho}<0$, and $h(s)\leq\hat{h}(s)<1$ for any sufficiently small $s>0$. Under some mild assumption {on the data distribution}, one can check that $\liminf_{s\rightarrow\infty}h(s)>1$ and thus there exists a unique positive $\alpha$ such that $h(\alpha)=1$.

In the next section, we study the properties of the tail-index of DE-SGD iterations further
\section{Theoretical Analysis for the Tail-Index}\label{sec:theoretical}

\textbf{4.1 Monotonicity of the tail-index.}
Assume that $\hat{\rho}<0$ so that there exists
a unique positive $\hat{\alpha}$ such that $\hat{h}(\hat{\alpha})=1$.
We can obtain the following monotonicity result for $\hat{\alpha}$ with respect to the hyperparameters of the model
by extending the proof of Theorem~4 in \citet{ht_sgd_quad} from centralized to the decentralized setting.

\begin{theorem}\label{thm:mono}
Suppose Assumptions \textbf{(A1)}-\textbf{(A2)} hold.
The tail-index $\hat{\alpha}$ is strictly increasing in batch-sizes $b_{i}$
and strictly decreasing in stepsize $\eta$ provided that $\hat{\alpha}\geq 1$.
Moreover, the tail-index $\hat{\alpha}$ is strictly decreasing in dimension $d$.
\end{theorem}
In the next section, we compare the DE-SGD to Dis-SGD (where nodes do not communicate at all with each other) in terms of the tail-index. We will also compare with the C-SGD.

\textbf{4.2 Tail-index comparison between Disconnected SGD and Centralized SGD. } We start with defining properly what exactly we mean by Disconnected SGD and Centralized SGD iterations.

\textbf{{Disconnected SGD}.} Disconnected SGD (Dis-SGD) corresponds to the case $W=I$ (where nodes do not share information with other nodes), and for every $i=1,2,\ldots,N$,
the iterates follow the recursion:
$x_{i}^{(k+1)}=x_{i}^{(k)}-\eta\tilde{\nabla}f_{i}(x_{i}^{(k)})$,
where each gradient $\tilde{\nabla}f_{i}(x_{i}^{(k)})$ is based on $b_{i}$ samples from node $i$'s dataset. The total number of samples (cumulatively over the nodes) equals $\sum_{i=1}^N b_i$.

\textbf{{Centralized SGD}.} Centralized SGD (C-SGD) consists of the iterations
$x_{k+1}=x_{k}-\eta\tilde{\nabla}f(x_{k})$,
where we take
(number of data points per iteration) batch-size
to be $\sum_{i=1}^{N}b_{i}$ for centralized SGD.

We will first be comparing C-SGD to Dis-SGD and then we will be comparing it to the DE-SGD.

To make the comparison of Dis-SGD and C-SGD easier, we assume
that $b_{i}\equiv b$ and $a_{i,j}$ are i.i.d. over $i$ and $j$.
Under Assumptions \textbf{(A1)}-\textbf{(A2)},
the iterates $x_{i}^{(k)}$ are independent and
as it is shown in \citet{ht_sgd_quad} that
$\hat{\alpha}=\hat{\alpha}(b)$ is a lower bound
of the tail-index of $x_{i}^{(\infty)}$
which is the unique positive value satisfying $\hat{h}(\hat{\alpha}(b))=1$,
where $\hat{h}(s)=\mathbb{E}[\Vert I-\eta H\Vert^{s}]$,
where $\hat{\alpha}(b)$ emphasize the dependence on the batch-size $b$
such that for each node $i$, $b$ data points are chosen.
We use $\hat{\alpha}(b)$ as a proxy of the tail-index.
In the centralized setting,
where $bN$ data points are chosen
at each iteration, and hence the batch-size
equals $bN$ for centralized SGD.
Thus, as it is shown in \citet{ht_sgd_quad} that
the corresponding tail-index (proxy) is $\hat{\alpha}(bN)$.
We have the following observation by adapting the
the monotonicity properties of tail-index shown in \cite[Theorem~4]{ht_sgd_quad}.

\begin{proposition}
\label{prop:disconnected:centralized}
The tail-index for disconnected SGD is smaller than that
of the centralized SGD. Indeed, their difference gets larger
as the network size increases.
\end{proposition}

\textbf{4.3 Tail-index comparison between Decentralized SGD and Disconnected SGD. }
We choose the mixing matrix as $W= I_N - \delta L$ with $\delta>0$ small enough so that the spectral radius of $W$ is not larger than 1, where we recall that $I_N$ denotes the $N\times N$ identity
matrix and $L$ is the $N\times N$ graph Laplacian. This choice of the mixing matrix has been common in the literature \citep{pu2020asymptotic, hendrikx2019accelerated}. We consider the function
$g(\delta) := \| \mathcal{W} - \eta H\| =
\| I_{Nd} - \eta H - \delta (L\otimes I_d) \|$, where we recall that $H^{(k+1)} := \mbox{blkdiag} ( \{ H_{i}^{(k+1)} \}_{i=1}^N )$ and $\mathcal{W} = W \otimes I_d$. Recall from \eqref{ineq-h-s} that $\hat{h}(s):=\mathbb{E}\left[\left(g(\delta)\right)^{s}\right]$ and the tail-index estimate $\hat\alpha = \hat\alpha (\delta)$ depends on $\delta$ and is the unique positive value that satisfies $\hat h\left(\hat\alpha(\delta)\right) = 1$ provided that $\hat{\rho}=\hat{\rho}(\delta)=\mathbb{E}[\log g(\delta)]<0$. The case $\delta = 0$ corresponds to Dis-SGD, whereas $\delta>0$ corresponds to the DE-SGD.
When $\delta=0$, we use the notation $\hat{h}_{dis}(s)=\mathbb{E}\left[\left(g(0)\right)^{s}\right]$, $\hat{\alpha}_{dis}=\hat{\alpha}(0)$ and $\hat{\rho}_{dis}=\hat{\rho}(0)$ for the disconnected case.
A natural question would be how the tail-index changes when the network effect is introduced; i.e. whether $\alpha'(0):=\frac{d\alpha(\sigma)}{d\sigma}|_{\sigma=0}$ is positive or not.
To answer this question, we use the Taylor
series expansion of $g(\delta)$ around $\delta=0$ by the standard perturbation theory for singular values
(see e.g. \cite[Lemma 2.3]{guglielmi2011fast})
to obtain a first-order expansion for $\hat{h}(s)$
in terms of $\hat{h}_{dis}(s)$, a correction term
linear in $\delta$, and a higher-order error term
(see Lemma~\ref{lem:first:order} in the Appendix).
Based on this, we can obtain a first-order approximation of $\hat{\alpha}$
in terms of $\hat{\alpha}_{dis}$, a correction term linear in $\delta$
and a higher-order error term, where the tail-indexes $\hat{\alpha}$ and $\hat{\alpha}_{dis}$
are the unique positive values such that $\hat{h}(\hat{\alpha})=1$
and $\hat{h}_{dis}(\hat{\alpha}_{dis})=1$ provided that $\hat{\rho}_{dis}<0$
and $\delta$ is sufficiently small.

\begin{theorem}\label{thm:first:order:alpha}
Assume $d=1$ and $b=1$.
Also assume that $\hat{\rho}_{dis}=\mathbb{E}\left[\log(\max_{1\leq i\leq N}|1-\eta a_{i}^{2}|)\right]<0$.
As $\delta\rightarrow 0$, we have
\begin{align}\label{alpha:expansion}
\hat{\alpha}&=\hat{\alpha}_{dis}
-\frac{s\delta N^{-1}\sum_{i=1}^{N}L_{ii}[1-2\mathbb{P}(\min_{1\leq i\leq N}a_{i}^{2}+\max_{1\leq i\leq N}a_{i}^{2}<2/\eta)]}{\mathbb{E}[\log(\max_{1\leq i\leq N}|1-\eta a_{i}^{2}|)
\max_{1\leq i\leq N}|1-\eta a_{i}^{2}|^{\hat{\alpha}_{dis}}]}+o(\delta),
\end{align}
where we provide an explicit formula for the probability and the expectation on the right hand side of \eqref{alpha:expansion} in Theorem~\ref{thm:first:order:alpha-extended} in the Appendix.
\end{theorem}

\begin{wrapfigure}{r}{0.5\textwidth}
\vspace{-0.35in}
  \begin{center}
    \includegraphics[width=0.4\columnwidth]{figures/probablity-contour.png}
    \label{fig:probability-contour}
  \end{center}
  \vspace{-0.2in}
  \caption{The contour plot of $e({\eta,N})=1-2\mathbb{P}\left(\min_{1\leq i\leq N}a_{i}^{2}+\max_{1\leq i\leq N}a_{i}^{2}<2/\eta\right)$.}

\end{wrapfigure}

We can see from Theorem~\ref{thm:first:order:alpha}
that the second term on the right hand side of \eqref{alpha:expansion}
is positive if and only if $e({\eta,N}):= 1 - 2\mathbb{P}\left(\min_{1\leq i\leq N}a_{i}^{2}+\max_{1\leq i\leq N}a_{i}^{2}<\frac{2}{\eta}\right)>0$,
and this term is monotonically increasing in $N$ and $\eta$.
Therefore, we conclude from Theorem~\ref{thm:first:order:alpha}
that when $\delta$ is small,
$\hat{\alpha}<\hat{\alpha}_{dis}$ given the stepsize $\eta$ or network size $N$
is large.
On the other hand, when $\delta$ is small,
$\hat{\alpha}>\hat{\alpha}_{dis}$ given the stepsize $\eta$ or network size $N$
is small. This is illustrated in Fig. \ref{fig:probability-contour} for $d=b=1$ and $a_i \sim \mathcal{N}(0,1)$ which shows the contour plot of $e(\eta,N)$, where the curve $e(\eta,N)=0$ is drawn in red color. In the plot, every$(\eta,N)$ pair satisfies $\hat\rho_{dis}<0$ and large stepsize or large enough network to the right of red curve results in heavier tail ($\hat\alpha<\hat\alpha_{dis}$) in DE-SGD.
We recall that $\hat{\alpha}$ and $\hat{\alpha}_{dis}$ are the unique positive values
such that $\hat{h}(\hat{\alpha})=1$ and $\hat{h}_{dis}(\hat{\alpha}_{dis})=1$.
They correspond to the tail-indexes in the decentralized case with network effect
and without network effect. We also have the following result that compares $\hat{\alpha}$ to  $\hat{\alpha}_{dis}$.

\begin{corollary}\label{cor:first:order:comparison}
(i) Under the assumptions in Theorem~\ref{thm:first:order:alpha},
further assume that the stepsize $\eta>\frac{2}{F_{a}^{-1}(2^{-\frac{1}{N}})}$.
For sufficiently small $\delta$, $\hat{\alpha}<\hat{\alpha}_{dis}$, i.e.
the tail gets heavier with the presence of network effect.
(ii) Further assume that $a_{i}$ are i.i.d. $\mathcal{N}(0,\sigma^{2})$ distributed.
Then, the first-order expansion \eqref{alpha:expansion} can be further simplified (see Corollary~\ref{cor:first:order:comparison:extended} in the Appendix).
Moreover, when the variance $\sigma^{2}>\frac{1}{\eta}(\text{erf}^{-1}(2^{-\frac{1}{N}}))^{-2}$,
for sufficiently small $\delta$, $\hat{\alpha}<\hat{\alpha}_{dis}$, i.e.
the tail gets heavier with the presence of network effect.
\end{corollary}
\begin{wrapfigure}{r}{0.5\textwidth}
  \begin{center}
    \includegraphics[width=0.4\columnwidth]{figures/probablity-contour-100.png}
  \end{center}

  \caption{\label{fig:probability-contour-100}The red curve satisfies $\hat\alpha=\hat\alpha_{dis}$ (up to a $o(\delta)$ term), $d$=$b$=100, $a_i \sim \mathcal{N}(0, I_d)$ with $\hat\rho_{dis}=0$ line in orange.}

\end{wrapfigure}
We can see from Cor.~\ref{cor:first:order:comparison} that when
the network size $N$ or the stepsize $\eta$ is large, then,
for sufficiently small $\delta$,
the tail gets heavier with the presence of network effect provided that $\hat{\rho}_{dis}<0$ (this condition ensures existence of stationary distribution and is tight in the sense that $\hat{\rho}_{dis}>0$ implies divergence \citep{buraczewski2016stochastic}.
Moreover, when $a_{i}$ are i.i.d. $\mathcal{N}(0,\sigma^{2})$ distributed
with $\sigma^{2}$ being large, then for sufficiently small $\delta$,
the tail gets heavier with the presence of network effect.
The general $d$ case is analyzed in the Appendix, and is illustrated in Fig. \ref{fig:probability-contour-100} for $d=100$. Similar to Fig. \ref{fig:probability-contour}, we can identify a red curve, which divides the $(\eta, N)$ space into two regions. In Region I (left-hand side of the red curve), DE-SGD has lighter tails compared to Dis-SGD, and in Region II (right-hand side of the red curve), DE-SGD has heavier tails compared to Dis-SGD. One difference with respect to the $d=1$ case, is that here the stepsize and $N$ cannot be too large otherwise the iterates will diverge. This is illustrated by the orange line which shows the points with $\hat\rho_{dis}=0$. In Region II, we have unstability $\hat\rho_{dis}>0$ (so that iterates diverge). Correspondingly, only part of the Region I is stable (with convergent iterates) and in this case DE-SGD has lighter tails.\looseness=-1

\textbf{4.4 Tail-index comparison between Decentralized SGD and Centralized SGD. }

In this section, we are interested in comparing the tail-index
for the DE-SGD with that of the C-SGD.
To make the comparison simpler, we assume $d=1$, $b_{i}\equiv 1$ and $\sigma_{i}\equiv\sigma$.
The general case will be provided in the Appendix.
In Proposition~\ref{prop:disconnected:centralized}, we showed that
the tail-index for Dis-SGD is smaller than that
of the C-SGD.
In Theorem~\ref{thm:first:order:alpha}
we showed that when $\delta$ is small,
the tail-index for the DE-SGD is smaller
than that of the Dis-SGD given the stepsize $\eta$ or network size $N$
is large.
Therefore, we have the following corollary
that compares DE-SGD with C-SGD.

\begin{corollary}\label{cor:DSGD:CSGD}
In the setting of Theorem~\ref{thm:first:order:alpha},
the tail-index $\hat\alpha$ of the decentralized SGD is smaller
than that of the centralized SGD provided that the stepsize $\eta$ or network size $N$
is large and $\delta$ is small.
\end{corollary}

On the other hand, we showed in Theorem~\ref{thm:first:order:alpha} when $\delta$ is small,
the tail-index for the DE-SGD is larger
than that of the Dis-SGD given the stepsize $\eta$ or network size $N$ is small.
Therefore, our theory (Proposition~\ref{prop:disconnected:centralized} and Theorem~\ref{thm:first:order:alpha})
does not provide a guidance to the comparison of the tail-indexes
between DE-SGD and C-SGD in this regime. However, in the numerical experiments section, we have always observed that DE-SGD iterates had heavier tails than C-SGD.

\section{Numerical Experiments}
We present our numerical results on both synthetic and real data in this section. Our main goal is to illustrate that DE-SGD iterates admit heavy-tailed distributions in the limit with proper choice of stepsizes $\eta$ and batch-sizes $b$. We will also validate that the behavior of the tail-index in DE-SGD is heavier than that of C-SGD under the assumptions we studied in Section~\ref{sec:theoretical} illustrating our theory. During the experiments, we found that the tail-index from different nodes are very close to each other. For better illustration, we will not report the tail-index of each local node; instead, we regard the median of $N$ tail indices from each node as the tail-index of the DE-SGD algorithm.

\begin{figure}[t]
\centering

    \hfill
    \subfigure[Case I: $d$=$b$=1, $\sigma$=1, $\sigma_y$=0.2, $N$=30 over complete network.]{
    \includegraphics[width=0.315\columnwidth]{figures/eta_tau_eta_d1.png}
    \label{fig:tau_eta_eta}
    }
    \hfill
    \subfigure[Case II: $d$=$b$=100, $\sigma$=1, $\sigma_y$=0.2, $N$=10 over complete network.]{
    \includegraphics[width=0.315\columnwidth]{figures/eta_tau_eta_10nodes_new.png}
    \label{fig:eta_tau_eta_d=1}
    }
    \hfill
    \subfigure[Case III: $d$=100, $b$=5, $\sigma$=1, $\sigma_y$=3, $N$=8 over star network.]{
    \includegraphics[width=0.315\columnwidth]{figures/lin-reg-delta-case3.png}
    \label{fig:eta_eta_tau}
    }

    \caption{Illustration of three main cases on synthetic data}

\end{figure}
\paragraph{5.1 Synthetic experiments.}
In this set of experiments, we consider a simple synthetic setup, where the studied problem is a simple linear regression and each data point is assumed to follow a Gaussian distribution. More specifically, we consider the following model:
\begin{equation}
     a_i \sim \mathcal{N}\left(0,\sigma^2I\right), \quad\epsilon \sim \mathcal{N} \left(0,\sigma_y^2\right), \quad y_i = x^T a_i + \epsilon,
\label{eqn:data_generation}
\end{equation}
where subscript $i$ indicates the node index,  $x, a_i \in \mathbb{R}^d$, $y_i \in \mathbb{R}$ for $i=1,2,\dots,n$ and $\sigma, \sigma_y > 0$.

\textbf{Tail index estimation.} By the generalized CLT result we obtain in the Appendix, the ergodic average of the iterates converges to an $\alpha$-stable distribution: $\alpha \in (0,2)$ in the case of a stationary distribution with an infinite variance and $\alpha=2$ otherwise. In our experiments, we take the average of the DE-SGD iterates and this enables us to advanced estimators \citep{mohammadi2015estimating} specific to $\alpha$-stable distributions, details are in the Appendix.

\paragraph{{Illustrating our Thm. \ref{thm:first:order:alpha} and Cor. \ref{cor:first:order:comparison}}.} Dis-SGD can be analyzed with the existing theory developed for C-SGD. In particular, by the results of \citet{ht_sgd_quad}, there exists an explicit constant $\eta_{\max}$ such that $\hat\rho_{dis}<0$ is equivalent to $\eta \in (0, \eta_{max})$ and for $\eta > \eta_{max}$ the distribution of the iterates diverge. Furthermore, Dis-SGD has a stationary distribution with heavy tails with infinite variance (with $\alpha<2$) only if $\eta \in (\eta_{crit}, \eta_{max})$; otherwise for $\eta \in (0, \eta_{crit}]$ the tail-index is greater than or equal to 2.
Theorem~\ref{thm:first:order:alpha} and Corollary~\ref{cor:first:order:comparison} show that provided that $\hat\rho_{dis}<0$ (i.e. provided that $0 < \eta < \eta_{max}$ ) when stepsize $\eta > \tau := \frac{2}{F_{a}^{-1}(2^{-\frac{1}{N}})}$, the tail gets heavier with the presence of network effect $\delta$, and when $\eta < \tau$, the tail gets lighter with the presence of network effect. If $\eta>\eta_{max}$, then we have divergence. Therefore, there are three main cases depending on the values of $\tau, {\eta}_{crit}$ and ${\eta}_{max}$:

\textbf{\underline{\textit{Case I}}: ($0<\tau<{\eta}_{crit}<{\eta}_{max}$)} In this case, for $\eta \in (0,\tau)$, stationary distribution of Dis-SGD and DE-SGD has a finite variance, so the tail index of the averaged iterates is $\alpha=2$. As stepsize increases, the stationary distribution of both Dis-SGD and DE-SGD will have an infinite variance (with $\alpha<2$), eventually DE-SGD has heavier tails compared to Dis-SGD, due to Thm. \ref{thm:first:order:alpha} and Cor. \ref{cor:first:order:comparison}.
This is illustrated in Figure~\ref{fig:tau_eta_eta} on a complete network with $N=30$, $d=b=\sigma=1$ and $\sigma_y=0.2$.\looseness=-1

\textbf{\underline{\textit{Case II}}: ($0<{\eta}_{crit}<\tau<{\eta}_{max}$)}
For $\eta \in (0,\eta_{max})$, both Dis-SGD and DE-SGD are expected to converge to a heavy-tailed distribution. While in the small stepsize regime, Dis-SGD will have heavier tail, in the big stepsize regime, DE-SGD will have heavier tail. This is consistent with Thm. \ref{thm:first:order:alpha} and Cor. \ref{cor:first:order:comparison}. Figure~\ref{fig:eta_tau_eta_d=1} illustrates this on a complete network $N=10$, $d=b=100$, $\sigma=1$, $\sigma_y=0.2$.\looseness=-1

\textbf{\underline{\textit{Case III}}: ($0<{\eta}_{crit}<{\eta}_{max}<\tau$)}
In this case, both Dis-SGD and DE-SGD are expected to converge to a heavy-tailed distribution, while Dis-SGD will have heavier tail than DE-SGD. We can observe this phenomenon in Figure~\ref{fig:eta_eta_tau}. Note that although the limit distribution of DE-SGD will have lighter tails compared to Dis-SGD, it still lead to heavier tail compared to C-SGD.

\textbf{{Effect of parameters}.} In the Appendix, we investigate the tail-index $\alpha$ of the stationary distribution of DE-SGD
over different network topologies with varied stepsize $\eta$ and varied batch size $b_i = b$. We found that the tails were monotonic with respect to $\eta$ and $b$ as predicted by Theorem~\ref{thm:mono}. We also found that DE-SGD had heavier tails
compared to C-SGD in every case. This is also consistent with our Corollary~\ref{cor:DSGD:CSGD}.

\paragraph{5.2 Deep learning experiments.}
This subsection examines the heavy-tail index of the iterates that C-SGD and DE-SGD converge to for neural networks. Tail index is estimated in a similar way to the synthetic section (please see the Appendix for the details)
\paragraph{Fully-connected network on MNIST.}
As a first dataset/architecture, we validate our theory on a three-layer  fully-connected network with MNIST dataset \citep{deng2012mnist}. We calculate the tail-index of C-SGD as well as DE-SGD with different topologies over varied learning rates. In this experiment, we assume there are $N=8$ nodes in the network and batch size is set to $b=5$. The model is trained for 10K iterations and step size $\eta$ ranges from $10^{-4}$ to $7.5\times 10^{-2}$.
 Figure~\ref{fig:fcn-mnist} shows the tail-index of the parameters in the first layer (the results of other layers are similar).
From this figure, it is observed that DE-SGD will lead to a heavier tail than both Dis-SGD and C-SGD, the behavior is  consistent with Cor. \ref{cor:DSGD:CSGD}.
\begin{wrapfigure}{r}{0.7\textwidth}
\subfigure[FCN on MNIST.]{
    \includegraphics[width=0.315\columnwidth]{figures/MNIST-FCN-combine.png}
    \label{fig:fcn-mnist}
    }
    \subfigure[ResNet-20 on CIFAR10.]{
    \includegraphics[width=0.315\columnwidth]{figures/resnet_24n16b_4.png}
    \label{fig:resnet_24n16b}
    }

    \caption{Tail-index $\alpha$ for different setting on MNIST and CIFAR10.}

\end{wrapfigure}
\paragraph{ResNet-20 on CIFAR10.}
As a second (larger-scale) experiment, we evaluate our theory on ResNet-20 model \citep{he2016deep} with CIFAR10 dataset \citep{krizhevsky2009learning}.
We compare the heavy-tail index between C-SGD and DE-SGD over different network topologies
with $N=24$. Further details of the experimental setup can be found in the Appendix.
Figure~\ref{fig:resnet_24n16b} illustrates the tail-index comparison between C-SGD and DE-SGD. It is observed that DE-SGD for all networks is heavier tailed than C-SGD but lighter tailed than Dis-SGD.  In the Appendix, we also obtained similar results for $N=8$. These results are consistent with the behavior of Region I reported in Fig. \ref{fig:probability-contour-100}. We cannot tune stepsize $\eta$ larger than $0.2$ since it will significantly degrade the test accuracy of ResNet-20 and result in instability issues for the iterates. For this reason, we do not observe the behavior of Region II which requires large stepsizes.

\looseness=-1
\section{Conclusion}
Existing works about the heavy-tails for centralized SGD (C-SGD) do not apply to the decentralized setting. We provided theoretical and numerical results showing that heavier tails arise in DE-SGD setting compared to C-SGD. We also compare DE-SGD to disconnected SGD where nodes distribute the data but do not communicate. Our theory and experiments uncover an interesting interplay between the tails and the network structure: we identify two regimes of parameters (stepsize and network size), where DE-SGD
can have lighter or heavier tails than disconnected SGD depending on the regime. Finally, we provided numerical experiments on synthetic data and neural networks which illustrate the results. \looseness=-1

\section*{Acknowledgements}
We thank Dr. Rong Jin for the stimulating discussions and for all the helpful feedback about our paper. Mert G\"{u}rb\"{u}zbalaban and Yuanhan Hu's research are supported in part by the grants Office of Naval Research Award Number
N00014-21-1-2244, National Science Foundation (NSF)
CCF-1814888, NSF DMS-2053485.  Umut \c{S}im\c{s}ekli's research is supported by the French government under management of Agence Nationale de la Recherche as part of the ``Investissements d'avenir'' program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute).
Lingjiong Zhu is grateful to the partial support from a Simons Foundation Collaboration Grant
and the grant NSF DMS-2053454 from the National Science Foundation.

\bibliographystyle{plainnat}

redpajama_set_name: RedPajamaArXiv


text:
'use strict';
angular.module('todoApp')
    .controller('todoListCtrl', ['$scope', '$location', 'todoListSvc', 'msalAuthenticationService', function ($scope, $location, todoListSvc, msalService) {
        $scope.error = "";
        $scope.loadingMessage = "Loading...";
        $scope.todoList = null;
        $scope.editingInProgress = false;
        $scope.newTodoCaption = "";

        $scope.editInProgressTodo = {
            Description: "",
            ID: 0
        };

        $scope.isActive = function (viewLocation) {
            return viewLocation === $location.path();
        };

        $scope.populate = function () {
            todoListSvc.getItems().then(function (results) {
                $scope.todoList = results.data;
                $scope.loadingMessage = "";
            }, function (err) {
                if(err.data)
                    $scope.error = err.data;
                $scope.loadingMessage = "";
            })
        };

        $scope.add = function () {

            todoListSvc.postItem({
                'title': $scope.newTodoCaption,
            }).then(function (results) {
                $scope.loadingMsg = "";
                $scope.newTodoCaption = "";
                $scope.populate();
            }, function (err) {
                $scope.error = err;
                $scope.loadingMsg = "";
            })
        };

        $scope.$on("msal:acquireTokenFailure", function (event, errorDesc, error) {
                if (error.indexOf("consent_required") !== -1 || error.indexOf("interaction_required") != -1 ) {
                    msalService.acquireTokenPopup(applicationConfig.webApiScopes).then(function (token) {
                    todoListSvc.getItems().then(function (results) {
                        $scope.toGoList = results;
                        $scope.loadingMessage = "";
                    }, function (err) {
                        console.log("error " + err);
                        $scope.loadingMessage = "";
                    });
                }, function (error) {
                });
            }

        });

        $scope.$on("msal:acquireTokenSuccess", function (event, tokenOut) {
        });

    }]);

redpajama_set_name: RedPajamaGithub


Curado Trust is a medium sized, flexible organization. Our management and staff are experienced professionals with a thorough understanding of the legal and administrative requirements for international companies. We are very dedicated to providing our clients with the personal and tailor made service that Curado Trust is known for.
Our client portfolio consists of internationally active companies and high net worth individuals from all over the world.


NEWARK, Ohio, June 22, 2018— Central Ohio Technical College (COTC) held a graduation ceremony on May 30, 2018, at the Newark campus to honor 21 graduates of the Peace Officer Basic Training (POBT) and name the recipients of the Officer Thomas W. Cottrell Jr. Award for Character and Dedication to Public Service.
The following students* graduated from BAS Class #17-050 under Commander Jeff Sowards. Students are grouped by hometown.
The following students* graduated from BAS Class #18-007 under Commander Jeff Sowards. Students are grouped by hometown.
From BAS Class #17-050, John Fridinger of Etna received the Officer Cottrell Award. Fridinger said he wanted to be a police officer because he wants to help the people who cannot help themselves. "When someone dials 9-1-1, it's because they need help right now, and I want to be the person who shows up to help," he stated.
Fridinger said he was very honored to receive the Officer Cottrell Award, and that it meant the world to him to be presented with it. He is presently going through the hiring process and is looking forward to getting started in his career as a police officer.
The Officer Cottrell Award recipient from BAS Class #18-007 was Jeremy Justice II of Thornville. Justice is in the United States Marine Corps so he wanted to continue a similar career. "The core values of honor, courage and commitment are important to me. My passion for helping people led me to pursue a career in law enforcement," he said.
His father is a Marine veteran and served 16 years as a police officer. His mother is a 17-year registered nurse who has specialized in trauma. "Their selfless and dedicated service inspired me to be the best I can be. I'm blessed to have my faith, family and friends' support as well as their encouragement throughout the academy. Our academy staff and instructors along with my fellow classmates have helped shape me into the man and law enforcement officer I want to be," he continued.
Justice will continue his education by pursuing degrees in law enforcement technology and criminal justice. He is presently going through the hiring process.
Officer Cottrell was a 2002 graduate of COTC's Peace Officer Basic Training. He was killed in the line of duty while working in the small Knox County town of Danville in January 2016. The Officer Cottrell Award is presented by the Cottrell family at each POBT graduation ceremony. It is the highest honor a cadet can receive. The selection of the recipients is based on many factors including, but not limited to, academic success and overall leadership and character. The $1,000 award assists the recipient with equipment purchases and other expenses associated with beginning a career in law enforcement.
The POBT prepares students to meet the requirements of the Ohio Peace Officer Training Commission. Graduates are recommended to take the state certification examination to become a peace officer in Ohio. Graduates' scores on the state certification exam ranked second in the state among open enrollment academies and fifth overall according to a 2016 report by the Ohio Attorney General's Office. Graduates also earn 25 credits toward an Associate of Applied Science in Law Enforcement Technology. For more information, visit cotc.edu/POBT.


text:
Tengku Zafrul
Rebuilding a Malaysia that Malaysians deserve
By Tengku Zafrul Abdul Aziz
By now, millions of Malaysians would have criss-crossed the country following the lifting of the inter-state travel ban on Monday. Some are out to fulfill their wanderlust but for many it is a long awaited heartfelt journey to meet their loved ones, perhaps for the first time after the ban was imposed earlier this year.
By any account, the lifting of the inter-state travel ban on Oct 11 marks an important turning point for Malaysia after nearly 20 months of battling an invisible enemy that has left a trail of public health and economic damage in its wake.
Yet, if there is anything that Malaysians are known for, it is our resilience. History has shown that we are hardwired to thrive amid any crisis. But will the pandemic make us stronger? Can we survive a repeat of this crisis? Herein lies our opportunity to ensure that this crisis does not go to waste.
Four key lessons from the pandemic
Prior to 2020, no country in this world had a pandemic playbook. Since last year, we have gathered not just knowledge about COVID-19, but also wisdom. The cold, brutal fatality of COVID-19 aside, we have warmed to the rich expressions of caring for and connection with our fellow human beings in equal measure.
Indeed, one silver lining to this pandemic is that it has forced us to be more strategic, creative and experimental. In fact, strategy, creativity and the courage to experiment are what we need to firmly set the nation on its healing journey towards the new normal. Related to this, I would like to share four key lessons that I feel represent the 'how' on healing, and on how we could be strategic and creative at the same time.
Lesson number one centers around our readiness for another pandemic. As a nation, we can be proud of having one of the world's fastest vaccination rates, but how do we enhance our resilience for future shocks? Covid-19 may not be the last pandemic to hit our shores, but we must make it the last time the country is caught in a zero-sum game of choosing between lives or livelihoods.
Can we use this crisis to fix our public health system, medical infrastructure and industrial capacity? Perhaps we can learn from South Korea, which was poorly prepared for SARS several years ago, but learnt fast such that when COVID-19 hit, the country was able to bounce back quickly. A few months back, the Operation Surge Capacity for the Klang Valley proved to be our commendable debut in COVID crisis response. It is exactly this capacity that we need to be able to scale up swiftly and effectively in the face of another crisis.
Lesson number two: it is crucial to go digital. Digitalization is a game-changer that can close the income gap between the haves and have-nots. During the various stages of lockdowns in the country since March 2020, we saw how tech grew as an economic enabler. Even those selling nasi lemak by the roadside have used socmed accounts to ply their trade online.
But we need to strengthen our nationwide digital backbone, so that even Malaysians in rural areas have access to tech-enabled opportunities. Our Digital Malaysia policy is set out to achieve this. We must embrace technology like 4IR, 5G, Big Data and Machine Learning, or risk being left behind in an increasingly tech-driven economy.
Lesson number three: let's build up our strategic supply chain. At the start of the crisis, we were struggling to procure masks, ventilators, respirators and other medical equipment required to save lives. If we are heavily reliant on imports, what will happen if borders are closed again? And what about our food security? This is also the best time to embed sustainability in all its forms—including social and environmental—into our supply chain principles because increasingly, sustainability and the bottom line are no longer mutually exclusive.
Lesson number four: this is an opportune time to strategically address the structural issues that have prevented us from realising our true socio-economic potential. The pandemic has highlighted many of the country's deep legacy faultlines. One of these is our job creation capacity. Since early 2021, through the National Employment Council (NEC), the Government has sought to create 500,000 new employment opportunities. In September, we successfully breached the 390,000-mark and God-willing we are on track to hit our target by December. Unemployment was at 5.3% in May 2020, but gradually reduced to 4.6% last August.
But the NEC goes beyond job creation. It is also strategic about job quality, and capacity building, like ensuring that graduates' skills match industry's needs; reskilling is provided to enable redundant jobbers to pivot to different industries; vocational education is properly developed; and entrepreneurship is continuously nurtured. This is the NEC's holistic strategy to build resilience in our people.
RMK12 and Budget 2022
Recently, the 12th Malaysia Plan (12MP) was unveiled, outlining the country's economic vision for the next five years. There are many goals reflected in the 12MP, but it is always the annual budget that brings to life the execution of our five-year plans at a more granular level.
On October 29, I will be tabling Budget 2022. As important as short-term policies are in driving our socio-economic recovery in 2022, it is equally critical that we pursue longer-term institutional and fiscal reforms, like the four lessons above, for rebuilding our resilience, so that we are better prepared to face any similar major setback in future.
Since the reopening of domestic and soon, international borders, our economic recovery has shifted to higher gear – and is poised for a speedy rebound. I am confident that as our recovery gathers momentum, we will come out of the pandemic stronger than before. We must ride on this momentum, and curate a better Malaysia that all Malaysians deserve.
(Tengku Zafrul bin Tengku Abdul Aziz is the Minister of Finance, Malaysia.)
Budget 2022: A color-blind budget for all in Keluarga Malaysia
Gearing up for Budget 2022


Sebastiania chamaelea Muell. Arg is a perennial herbal medicinal plant of family Euphorbiaceae and it is the only species of this genus represented in India. It is commonly found as forest undergrowth and also in cultivated lands. Based on folklore uses, the decoction of the plant in ghee is given as tonic and applied to the head in vertigo. Juice of the plant is astringent and is used as a remedy for syphilis and diarrhoea. Following the above folk claims for cure of vertigo, syphilis and diarrhoea, phytochemical and biological evaluation was carried out to test efficacy of the plant scientifically. Based on ethnobotanical data obtained from Nigerien and Senegalense traditional healers S.chamaelea istraditionally used to treat malaria. The studies revealed presence of various phytochemical compounds and some notable \ pharmacological activities of the plant such as antibacterial, antifungal, antioxidant, anthelmintic, antidiarrhoeal and antidiabetic activities, presented in this review such that the potential use of the plant in pharmaceutics can be evaluated. The present review is an attempt to highlight phytochemicals, pharmacolological as well as tissue culture reports on Sebastiania chamaelea Muell. Arg.
Keywords: Sebastiania chamaelea, Phytochemicals, Folklore uses, Pharmacological activity.


A sprinkler irrigation system typically comprises of items such as water supply, pumps and distribution system.
Complete this electronic spreadsheet to ensure that your Sprinkler irrigation systems product information meets the requirements of Level 2 BIM. This is important as it will enable your customers to select, specify and use your products within the BIM environment. Once you have completed this template you can host it on your own website or distribute it to your customers. Please note that we do not host completed product data templates within the BIM Toolkit.
The above illustration is for Sprinkler irrigation systems from the NBS section Irrigation systems. This is indicative of the LOD requirements for Sprinkler irrigation systems.
The above illustrations are for Sprinkler irrigation systems from the NBS section Irrigation systems. This is indicative of the LOD requirements for Sprinkler irrigation systems.
System manufacturer The System manufacturer of the Sprinkler irrigation systems.
Standard The Standard of the Sprinkler irrigation systems.
Water supply - Source The Water supply - Source of the Sprinkler irrigation systems.
Pumps Products such as Close-coupled end suction pumps.
Distribution system - Valves and controls - Air valves Products such as Automatic air vents.
Distribution system - Valves and controls - Riser valves Products such as Waterworks gate valves.
Distribution system - Valves and controls - Surge control Products such as Control valves.
Distribution system - Valves and controls - Pressure regulation Products such as Pressure reducing valves.
Distribution system - Valves and controls - Zone control Products such as Control valves.
Distribution system - Valves and controls - Volumetric control Products such as Control valves.
Distribution system - Strainers and filters Products such as Dirt separators.
Distribution system - Controls Products such as Irrigation control panels.
Application devices - Lateral pipelines Products such as Copper pipelines.
Application devices - Outlets Products such as Rain guns.
System accessories Products such as Anemometers.


Things took a more serious and scientific turn at a recent Friday Seminar, Life Chiropractic College West's weekly all-student meeting.
While most presentations are around student body business, such as student council elections or awards, or strike a more inspirational tone, such as a recent talk from Life West President Dr. Ron Oberstein, this one was notable for featuring Dr. Amy Haas, a New Hampshire chiropractor who is also an expert on chiropractic and heart rate variability, or HRV.
Dr. Amy Haas speaks to Life West students recently.
Dr. Haas, who runs her own practice in New Hampshire and gave several examples taken from her own patients, gave a presentation called "Why HRV? Creating Value with Certainty in Chiropractic Practice." Her lecture, which she had given to faculty members earlier, frames HRV as a unique opportunity for the chiropractic profession. She says that being able to measure HRV allows chiropractors to differentiate themselves from other healthcare disciplines, and to promote the core chiropractic value of improving adaptability as modulated by innate intelligence.
During the presentation, Dr. Haas drew connections for the audience between chaos theory, how the human body works and the patterns that are created by the nervous system, and heart rate variability. Medical findings show that increased heart rate variability leads to better health, and she has documented among her own patients a link between chiropractic adjustments for subluxation and increased HRV.
Dr. Haas said that making adjustments for subluxation allows for better sensory input into the nervous system, which fits into the chiropractic idea of innate intelligence. Innate intelligence says that the body knows how to heal itself and can, if given the right circumstances. One example of that is the way in which a broken bone knits properly, once it is set.
Dr. Haas has published an open-access article about her work, "Sustained improvement of heart rate variability in patients undergoing a program of chiropractic care: A retrospective case series" in Chiropractic Journal Australia 2018. It provides ideas for further research but is not proof of the relationships in itself.
Watch her talk about HRV as well in this Mile High Chiro podcast with Dr. Daniel Knowles.


text:
Build your meals
At Marleybones, health and earth go hand-in-hand.
We all want to do what is best for our four-legged friends. But we have all seen images of oceans and beaches filled with plastic. We have all seen chickens cramped in cages.
It is time to think of the earth outside the comfort of our home. At Marleybones, we aim to create the pawfect meals, with every step of the supply chain being as sustainable as it is delicious.
This does good for your dog, our planet and you.
The UN's Sustainable Development Goals
At Marleybones we are committed to supporting the UN's Sustainable Development Goals (SDGs).
Divided across three pillars of animal welfare, environment and people, we are working to tackle key environmental challenges, create healthier and more nutritious pet food, help less fortunate street dogs and use innovative technological solutions to create awareness and focus on what's matters.
The earth.
three main priorities
We have organised our priorities in three categories;
animal welfare, environment and people.
Our approach to sustainability goes beyond our own operations and customers. With the implementation of blockchain, we ensure our entire value chain lives up to the highest welfare and sustainability standards – open for everyone to see.
We recognised that we can only create a more sustainable world by including the interconnected nature of health, environmental and social challenges we face. We work to maximise sustainable values for our customers, business and communities.
01. animal welfare
Animal welfare and ethically sourced products is among our highest priorities. Sadly, intensive industrial farming is often associated with poor animal welfare and unjust conditions. Marleybones will not be a part of this practise under any circumstance.
We are committed to sourcing only free-range chickens, grass-fed lamb and cattle with the safest slaughtering practises, and healthy salmon from an all-natural, hand-rearing environment. As an added benefit our meat and fish are much higher in nutrients!
Close to our heart is the support of WECare Worldwide that neuter, vaccinate and treats street dogs round the clock.
Besides the greater environment, we are of course committed to keeping our pups safe by excluding all nasties from our meals, including sugar, artificial preservatives and flavours.
02. environment
We are comitted to a low-carbon circular economy in which the entire value chain has minimal climate impact. This includes reducing waste through prevention, reduction, recycling and reuse.
We have carefully selected British suppliers that are equally engaged in protecting our environment, by covering their fields with solar panels or renewing food waste into biomass or fertiliser, to name a few. With 98% of our produce being grown or reared within the Great British borders, we've ensured lower food miles when serving dog dining across the country.
Our meals are delivered in packaging made from sustainable sourced, plant-based renewable materials and are recyclable. Recycling is at the centre of a low-carbon circular economy that aims to reduce waste and keep materials in use for longer.
With the implementation of the blockchain technology from Provenance we ensure our entire value chain lives up to the highest welfare and sustainability standards – open for everyone to see.
Tetra Pak cartons are widely recyclable within the UK. Compared to the widely used can, our packaging takes up much less space, meaning less lorries on the road, contributing to a smaller carbon footprint. From production to waste management, they have even shown to have an overall climate impact five times less than steel or glass. As an added bonus, they are easy to store and take up little space!
More than 94% of councils in the UK collect the cartons and at Tetra Pak's own recycling plant the cartons are remade into new products like roofing tiles and toilet paper roll cores.
Find your nearest recycling facility here
Recycling collections
03. People
Behind Marleybones is two Danish female founders and owners and whole lot of other inspiring and hard working people.
Currently in the UK, less than 1% of venture capital investments go to startups with female founders while mixed gender startups receive 10% and male founders receive 89%. This is the kind of inequality gap that needs changing!
We believe diversity yields the best results and that is why we have implemented an Equal opportunities policy only to name a few. We do not discriminate on the basis of race, colour, nationality, sex or religion to name a few. In fact, we promote social, economical and political inclusion and support gender equality to its fullest.
We further make sure everyone working at Marleybones receives a fair payment.
Go green with
Marleybones
Start building your meal
Terms of use Terms & conditions Privacy policy
hello@marleybones.com Drop us a DM Send us a message
Meals+


CHANTILLY, Va. (AP) ¿ Financial technology company Online Resources Corp. on Tuesday issued a fourth-quarter revenue outlook below Wall Street expectations and said Matthew P. Lawlor has retired as CEO, effective immediately.
Lawlor will remain chairman until Feb. 15 to assist with the transition, the company said, and after that time, Lawlor will serve on the board as a director.
Online Resources said Raymond T. Crosier, 54, will serve as interim CEO until a successor is named. Crosier is currently president and chief operating officer.
The company also forecast revenue between $37 million and $37.5 million for the fourth quarter ending Dec. 31. That's below the $37.8 million estimate expected by analysts polled by Thomson Reuters.
Online Resources said its loss available to common shareholders is expected to range between a 3-cent loss and 2-cent loss, or between 9 cents and 10 cents in earnings when excluding certain items.
Analysts, who usually exclude one-time items, expect 10 cents per share in earnings.
The company expects to report fourth-quarter and full-year 2009 results in February. It said the fourth-quarter outlook doesn't include one-time items related to Lawlor's departure.
Online Resources said it formed an executive management committee consisting of Crosier, lead independent director Barry D. Wessler and independent director John Dorman to oversee the CEO transition.
Wessler and Dorman will lead the search for a new CEO, and the company expects to hire a firm to help identify CEO candidates.


Name the greatest example you have witnessed in maintaining confidence through adversity - past or present.
after watching the video, i've forgotten the question! have to go back and read. hard to beat the example set by Kyle. he is truly amazing! however i would like to suggest my daughter-in -law, Jill, a stay-at-home mother of six, one with severe brain damage. although her days and some nights are quite challenging, she most always smiles and counts each day with joy. tha joy of the Lord is litterally her strength! she truly is an example in our family of perseverence.
I have to give credit to my own mom, because she's faced more adversity than most people I know, and although her confidence does waver, she always comes back to centre - somehow. My mother has been sober for 26 years and is still committed to her sobriety every single day. I do give her a hard time for her eccentricities, but I really value her commitment to sobriety and the confidence with which she commits to her sobriety.


text:
UI joins IIE Scholar Rescue Fund Alliance in support of threatened and displaced scholars
By: International Programs | 2021.03.24 | 09:41 am
The University of Iowa has joined the Institute of International Education Scholar Rescue Fund (IIE-SRF) Alliance, a global network of partners offering practical support to scholars threatened by war and persecution.
The IIE-SRF Alliance launched in August 2020 to assist the unprecedented number of threatened and displaced scholars globally. It builds upon IIE's 100 years of providing resources and support to vulnerable professors and university students, including academics facing Nazism and Fascism, Black South Africans being denied educational opportunities under apartheid, and hundreds of scholars who are continually threatened by conflict and repression.
"The University of Iowa is proud to support the IIE-SRF alliance," says UI President Bruce Harreld. "Freedom of inquiry and academic freedom are critical cornerstones of a healthy society that must be protected at all costs. When one of our colleagues needs us, we will always provide a safe haven."
While academics historically have been persecuted for their scholarly work and are among the first to be targeted during conflict and instability, the magnitude of the current crisis is unprecedented, says Allan Goodman, IIE's president and CEO.
"We are extremely grateful to the University of Iowa for taking this concrete step to support scholars in need," says Goodman.
This effort is not new or unfamiliar to the UI, however. The UI's International Writing Program (IWP) has a long history of hosting writers and scholars at risk.
"The International Writing Program has, from its beginning in 1967, provided refuge for writers from all around the world. We welcome this alliance with IIE-SRF, which could not come at a more important moment with authoritarian governments on the march and scholars and writers in ever-increasing danger," says IWP Director Christopher Merrill. "Truthtellers are always a threat to those who seek to oppress their citizens, which is why we are committed to advocating for those who are determined to tell the truth. This alliance gives us new tools to keep the truthtellers alive."
The IWP and UI's International Programs previously worked with the IIE, bringing a writer at risk from Uzbekistan, Sabit Madaliev, to the UI in 2004. At the time, the prominent Uzbek poet and journalist had published numerous articles on sensitive topics of human rights, freedom of expression, freedom of information, corruption, and democracy in Central Asia, as well as more than 14 selections of poetry and fiction in Russian. He had been subjected to difficulties in his home country because he was an outspoken defender of other writers, journalists, and members of the media, as well as a member of the advisory committee for the Art and Culture Program of the Open Society Institute's branch in Tashkent, and a member of PEN International.
UI joins a network of hundreds of individuals, institutions, and organizations through the IIE-SRF Alliance, spanning nearly 30 countries and multiple sectors. The IIE-SRF Alliance includes:
Host and commitment partners that welcome IIE-SRF fellows to their campuses, providing safe haven and a temporary academic position.
Organizational partners that offer services such as access to training or courses, free or discounted memberships, and other critical support.
Ambassadors who share their time, skills, and networks to empower these scholars on their journeys to rebuild their lives and careers.
"The University of Iowa's participation in the Institute of International Education Scholar Rescue Fund Alliance demonstrates the UI's commitment to scholars whose lives and livelihoods have been threatened by upheaval across the globe," says Associate Provost and Dean of International Programs Russell Ganim. "As turmoil and displacement can take many forms, the goal in belonging to this organization is to show support for intellectuals, artists, and researchers whose valuable work has been disrupted by events beyond their control. The UI has provided a haven to these colleagues in the past, and in turn, they have enriched our community with their wisdom and insight regarding the cultures and contexts from which their work originates and evolves."
To learn more about IIE-SRF, the IIE-SRF Alliance, and ways to get involved in supporting threatened and displaced scholars, visit ScholarRescueFund.org.
Joan Kjaer, International Programs, 319-335-2026
News From:
News For:
News About:
Writing Arts
International Writing Program
International Education Scholar Rescue Fund Alliance
Links in this article are preserved for historical purposes, but the destination sources may have changed.


Kenson is the professional choice for —- for over 50 years. Life-like aesthetics, unparalleled value.
Myerson is proud to announce that Kenson Teeth, the professional's choice for 50 years, are now available in 16 matching VITA shades that are True to Life. Myerson continues to offer Kenson Teeth in the 10 original Kenson shades: 55, 61, 62, 65, 66, 67, 69, 77, 81, 87, plus New whiter white shade 51.
Time proven formula reinforced with sub-micron silica to enhance hardness.
Premium quality teeth hand-crafted to be virtually indistinguishable from natural teeth.


The end of 2014 is near….
So 2014 had a lot going on and ohiophotogrl and I are so glad that you are hanging out with us and watching what we get into.
In 2014 our blog was viewed 1700 times by great people from all over the world. And we have 192 followers and are looking forward to hitting 200 really soon!
We are looking forward to a great 2015!
We went out early today and stopped in the Schuster Center in Dayton, Oh to see the Rikes Window displays again.
We met our Pal ScottIeScouts there before we had lunch.
I could watch these displays all day.
Here is some more info on the displays ….
Well I'm off to figure out what I'm going to do to ring in the new year!
Ohiophotogrl isn't feeling so hot today so I am going to take care of her.
I made myself a mask so I can't catch what ever it is she has. I have been bringing her meds when she needs them and keeping her all covered up. She says I am a good nurse. Maybe that is what I will be when I grow up. I do like to help people.
Today we celebrated Christmas with Ohiophotgrl's family. What fun we had.
Then we loaded up the car.
We saw ohiophotgrls niece today too.
I am blessed to be apart of such a great family. They let me take a picture of them to share with you….
Hope you have a treat holiday season!
Today is Ohiophotogrls dads bday!
We went out to breakfast with with him and her nieces.
I had a yummy omlet and Ohiophotogrl had some potatoes with it.
Good time was had by all!
We have been enjoying making cookies for Ohiophotgrls family.
Then we mixed a few others and came up with these….
Boy oh boy did that keep us busy for a few days!
In 2013 we blogged everyday for a whole year. Last year we didn't do so hot in keeping you all updated on our adventures so we are going to step it up. Thats right Ohiophotgrl and I are gonna spend some quality time doing things to share with you. We may not see you everyday but you are going to see a WHOLE lot more of us!
with that being said here are a few of our favorite times this last year.
Ohiophotogrl and I had to take a road trip. Her uncle passed away.
The good news is she did get to see family she hasn't seen in a long time, the bad news is her We won't get to spend time with everyone we want to see. away. Cahokia Illinois here we are.
So I will say a little prayer for her and her family. For peace and safe travel.


As the massive forest fires continue to decimate parts of La Palma in the Spanish Canaries, some 600 firefighters are trying their hardest to douse the raring flames to no avail. Several thousand people have already evacuated.
Just a few thousand miles away however, another fire rages. Deep in the hearts of many committed poker players, the WSOP 2009 is celebrating its 40th anniversary and as we saw at the end of Day 8, it is only a matter of time before we reach the end of this world famous competition.
Day 8 saw a number of people get incinerated, and with only 9 players surviving, the final table has been made as these guys wait patiently for the last game to be held November 7 – 10.
But just what has happened in the past forty years that made the WSOP the defining competition in the global poker tournament calendar? Unlike bush fires, the biggest thing to redefine the poker game experience was not firemen, but the age of the online poker room and the rise of unknown players into the realms of casino stardom.
Back in 2003, it all started when a relatively unknown poker player won his bracelet, and he not only made himself famous, but he proved on behalf of the world, that anyone can win the WSOP. But, just who was this guy? – Chris MoneyMaker, the small time accountant turned Pokerstars tournament player, who managed to turn a $39 sit-in to a $10,000 seat at the WSOP and then? He won. And in case you are wondering, that really IS his surname.
Nowadays there are countless players from the online community, who have turned their lives around by simply discovering the challenge of mastering online poker games. Many have said that they simply couldn't have reached where they are today were it not for the likes of poker bonuses and rakeback.
Why not follow in Chris's footsteps? Take advantage of rakeback so you too can learn how to master the millionaire game of Texas Hold'em. Who knows, you may be the next one to hit the WSOP!
Are You the Next Masked Master?
What does the Ronaldo PokerStars Signing mean for Poker?