text:
\section{Introduction}
\label{intro}
Human eye sees the world in an interesting way. We suppose as if we see the entire scene at once, but this is an illusion created by the subconscious part of our brain \cite{goodfellow2016}. According to the {\em Scanpath} theory \cite{noton1971,noton1971a}, when the human eye looks at an image, it can see only a small patch in high resolution. This small patch is called the \emph{fovea}. It can see the rest of the image in low resolution which is called the \emph{periphery}. To recognize the entire scene, the eye performs feature extraction based on the fovea. The eye is moved to different parts of the image until the information obtained from the fovea is sufficient for recognition \cite{alpaydin1996}. These eye movements are called \emph{saccades}. The eye makes successive fixations until the recognition task is complete. This sequential process happens so quickly that we feel as if it happens all at once.

Biologically, this is called \emph{visual attention system}. Visual attention is defined as the ability to dynamically restrict processing to a subset of the visual field \cite{ahmad1991}. It seeks answers for two main questions: \emph{What} and \emph{where} to look? Visual attention has been extensively studied in psychology and neuroscience; for reviews see \cite{posner1990,bundesen1990,desimone1995,corbetta2002,petersen2012}. Besides, there is a large amount of literature on modeling eye movements \cite{rimey1990,sheliga1994,sheliga1995,hoffman1995}. These studies have been a source of inspiration for many artificial intelligence tasks. It has been discovered that the attention idea is useful from image recognition to machine translation. Therefore, different types of attention mechanisms inspired from the human visual system have been developed for years. Since the success of deep neural networks has been at the forefront for these artificial intelligence tasks, these mechanisms have been integrated into neural networks for a long time.

This survey is about the journey of attention mechanisms used with neural networks. Researchers have been investigating ways to strengthen neural network architectures with attention mechanisms for many years. The primary aim of these studies is to reduce computational burden and to improve the model performance as well. Previous work reviewed the attention mechanisms from different perspectives \cite{chaudhari2019}, or examined them in context of natural language processing (NLP) \cite{galassi2019,lee2019a}. However, in this study, we examine the development of attention mechanisms over the years, and recent trends. We begin with the first attempts to integrate the visual attention idea to neural networks, and continue until the most modern neural networks armed with attention mechanisms. One of them is the \emph{Transformer}, which is used for many studies including the \emph{GPT-3} language model \cite{brown2020}, goes  beyond convolutions and recurrence by replacing them with only attention layers \cite{vaswani2017}. Finally, we discuss how much more can we move forward, and what's next?

\section{From the Late 1980s to Early 2010s: The Attention Awakens}
\label{sec:1}
The first attempts at adapting attention mechanisms to neural networks go back to the late 1980s. One of the early studies is the improved version of the \emph{Neocognitron} \cite{fukushima1980} with selective attention \cite{fukushima1987}. This study is then modified to recognize and segment connected characters in cursive handwriting \cite{fukushima1993}. Another study describes {\em VISIT}, a novel model that concentrates on its relationship to a number of visual areas of the brain \cite{ahmad1991}. Also, a novel architecture named {\em Signal Channelling Attentional Network (SCAN)} is presented for attentional scanning \cite{postma1997}.

Early work on improving the attention idea for neural networks includes a variety of tasks such as target detection \cite{schmidhuber1991}. In another study, a visual attention system extracts regions of interest by combining the bottom-up and top-down information from the image \cite{milanese1994}.  A recognition model based on selective attention which analyses only a small part of the image at each step, and combines results in time is described \cite{alpaydin1996}. Besides, a model based on the concept of selective tuning is proposed \cite{tsotsos1995}. As the years go by, several studies that use the attention idea in different ways have been presented for visual perception and recognition \cite{culhane1992,reisfeld1995,rybak1998,keller1999}.

By the 2000s, the studies on making attention mechanisms more useful for neural networks continued. In the early years, a model that integrates an attentional orienting \emph{where} pathway and an object recognition \emph{what} pathway is presented \cite{miau2001}. A computational model of human eye movements is proposed for an object class detection task \cite{Zhang2006}. A serial model is presented for visual pattern recognition gathering Markov models and neural networks with selective attention on the handwritten digit recognition and face recognition problems \cite{salah2002}. In that study, a neural network analyses image parts and generates posterior probabilities as observations to the Markov model. Also, attention idea is used for object recognition \cite{walther2002}, and the analysis of a scene \cite{schill2001}. An interesting study proposes to learn sequential attention in real-world visual object recognition using a Q-learner \cite{paletta2005}. Besides, a computational model of visual selective attention is described to automatically detect the most relevant parts of a color picture displayed on a television screen \cite{lemeur2006}. The attention idea is also used for identifying and tracking objects in multi-resolution digital video of partially cluttered environments \cite{gould2007}.

In 2010, the first implemented system inspired by the fovea of human retina was presented for image classification \cite{larochelle2010}. This system jointly trains a restricted Boltzmann machine (RBM) and an attentional component called {\em the fixation controller}. Similarly, a novel attentional model is implemented for simultaneous object tracking and recognition that is driven by gaze data \cite{bazzani2011}. By taking advantage of reinforcement learning, a novel recurrent neural network (RNN) is described for image classification \cite{mnih2014}. \emph {Deep Attention Selective Network (DasNet)}, a deep neural network with feedback connections that are learned through reinforcement learning to direct selective attention to certain features extracted from images, is presented \cite{stollenga2014}. Additionally, a deep learning based framework using attention has been proposed for generative modeling \cite{tang2014}.
\section{2015: The Rise of Attention}
\label{sec:2}
It can be said that 2015 is the golden year of attention mechanisms. Because the number of attention studies has grown like an avalanche after three main studies presented in that year. The first one proposed a novel approach for neural machine translation (NMT) \cite{bahdanau2015}. As it is known, most of the NMT models belong to a family of encoder-decoders \cite{sutskever2014,cho2014a}, with an encoder and a decoder for each language. However, compressing all the necessary information of a source sentence into a fixed-length vector is an important disadvantage of this encoder-decoder approach. This usually makes it difficult for the neural network to capture all the semantic details of a very long sentence \cite{goodfellow2016}.

\begin{figure}[h!]
\centering
\includegraphics[scale=.5]{encoder_decoder}
\caption{The extension to the conventional NMT models that is proposed by \cite{bahdanau2015}. It generates the \emph{t}-th target word $y_t$ given a source sentence $(x_1, x_2, ..., x_T)$. }
\label{encoder_decoder}
\end{figure}

The idea that \cite{bahdanau2015} introduced is an extension to the conventional NMT models. This extension is composed of an encoder and decoder as shown in Fig \ref{encoder_decoder}. The first part, encoder, is a {\em bidirectional RNN} (BiRNN) \cite{schuster1997} that takes word vectors as input. The forward and backward states of BiRNN are computed. Then, an \emph{annotation} $a_j$ for each word $x_j$ is obtained by concatenating these forward and backward hidden states. Thus, the encoder maps the input sentence to a sequence of annotations $(a_1,...,a_{T_x})$. By using a BiRNN rather than conventional RNN, the annotation of each word can summarize both the preceding words and the following words. Besides, the annotation $a_j$ can focus on the words around $x_j$ because of the inherent nature of RNNs that representing recent inputs better.

In decoder, a weight $\alpha_{ij}$ of each annotation $a_j$ is obtained by using its associated energy $e_{ij}$ that is computed by a feedforward neural network \emph{f} as in Eq. (\ref{bahdanauenergy}). This neural network \emph{f} is defined as an \emph{alignment model} that can be jointly trained with the proposed architecture. In order to reduce computational burden, a multilayer perceptron (MLP) with a single hidden layer is proposed as \emph{f}. This alignment model tells us about the relation between the inputs around position \emph{j} and the output at position \emph{i}. By this way, the decoder applies an attention mechanism. As it is seen in Eq. (\ref{bahdanaualpha}), the $\alpha_{ij}$ is the output of softmax function:

\begin{equation}
e_{ij} = f(h_{i-1},a_j)
\label{bahdanauenergy}
\end{equation}

\begin{equation}
\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x}\exp(e_{ik})}
\label{bahdanaualpha}
\end{equation}

Here, the probability $\alpha_{ij}$ determines the importance of annotation $a_j$ with respect to the previous hidden state $h_{i-1}$. Finally, the context vector $c_i$ is computed as a weighted sum of these annotations as follows \cite{bahdanau2015}:

\begin{equation}
c_i = \sum_{j=1}^{T_x} \alpha_{ij} a_j
\label{bahdanauattention}
\end{equation}

Based on the decoder state, the context and the last generated word, the target word $y_t$ is predicted. In order to generate a word in a translation, the model searches for the most relevant information in the source sentence to concentrate. When it finds the appropriate source positions, it makes the prediction. By this way, the input sentence is encoded into a sequence of vectors and a subset of these vectors is selected adaptively by the decoder that is relevant to predicting the target \cite{bahdanau2015}. Thus, it is no longer necessary to compress all the information of a source sentence into a fixed-length vector.

The second study is the first visual attention model in image captioning \cite{xu2015}. Different from the previous study \cite{bahdanau2015}, it uses a deep convolutional neural network (CNN) as an encoder. This architecture is an extension of the neural network \cite{vinyals2015} that encodes an image into a compact representation, followed by an RNN that generates a corresponding sentence. Here, the {\em annotation vectors} $a_i \in R^D$ are extracted from a lower convolutional layer, each of which is a \emph{D}-dimensional representation corresponding to a part of the image. Thus, the decoder selectively focuses on certain parts of an image by weighting a subset of all the feature vectors \cite{xu2015}. This extended architecture uses attention for salient features to dynamically come to the forefront instead of compressing the entire image into a static representation.

The context vector $c_t$ represents the relevant part of the input image at time \emph{t}. The weight $\alpha_i$ of each annotation vector is computed similar to Eq. (\ref{bahdanaualpha}), whereas its associated energy is computed similar to Eq. (\ref{bahdanauenergy}) by using an MLP conditioned on the previous hidden state $h_{t-1}$. The remarkable point of this study is a new mechanism  $\phi$ that computes $c_t$ from the annotation vectors $a_i$ corresponding to the features extracted at different image locations:

\begin{equation}
c_t = \phi(\big\{a_i\big\},\big\{\alpha_i\big\})
\end{equation}

The definition of the $\phi$ function causes two variants of attention mechanisms: The \emph{hard (stochastic)} attention mechanism is trainable by maximizing an approximate variational lower bound, i.e., by REINFORCE \cite{williams1992}. On the other side, the \emph{soft (deterministic)} attention mechanism is trainable by standard backpropagation methods. The hard attention defines a location variable $s_t$, and uses it to decide where to focus attention when generating the \emph{t-th} word. When the hard attention is applied, the attention locations are considered as intermediate latent variables. It assigns a multinoulli distribution parametrized by ${\alpha_i}$, and $c_t$ becomes a random variable. Here, $s_{t,i}$ is defined as a one-hot variable which is set to 1 if the \emph{i-th} location is used to extract visual features \cite{xu2015}:

\begin{equation}
p(s_{t,i} = 1 | s_{j<t}, a) = \alpha_{t,i}
\end{equation}

\begin{equation}
c_t = \sum_i s_{t,i} a_i
\end{equation}

Whereas learning hard attention requires sampling the attention location $s_t$ each time, the soft attention mechanism computes a weighted annotation vector similar to \cite{bahdanau2015} and takes the expectation of the context vector $c_t$ directly:

\begin{equation}
E_{p(s_t|\alpha)}[c_t] = \sum_{i=1}^L \alpha_{t,i} a_i
\end{equation}

Furthermore, in training the deterministic version of the model, an alternative method namely \emph{doubly stochastic attention}, is proposed with an additional constraint added to the training objective to encourage the model to pay equal attention to all parts of the image.

The third study should be emphasized presents two classes of attention mechanisms for NMT: the \emph{global} attention that always attends to all source words, and the \emph{local} attention that only looks at a subset of source words at a time \cite{luong2015}. These mechanisms derive the context vector $c_t$ in different ways: Whereas the global attention considers all the hidden states of the encoder, the local one selectively focuses on a small window of context. In global attention, a variable-length alignment vector is derived similar to Eq. (\ref{bahdanaualpha}). Here, the current target hidden state $h_t$ is compared with each source hidden state $\bar{h}_s$ by using a \emph{score} function instead of the associated energy $e_{ij}$. Thus, the alignment vector whose size equals the number of time steps on the source side is derived.
Given the alignment vector as weights, the context vector $c_t$ is computed as the weighted average over all the source hidden states. Here, \emph{score} is referred as a \emph{content-based} function, and three different alternatives are considered \cite{luong2015}.

On the other side, the local attention is differentiable. Firstly, an aligned position $p_t$ is generated for each target word at a time $t$. Then, a window centered around the source position $p_t$ is used to compute the context vector as a weighted average of the source hidden states within the window. The local attention selectively focuses on a small window of context, and obtains the alignment vector from the current target state $h_t$ and the source states $\bar{h}_s$ in the window \cite{luong2015}.

The introduction of these novel mechanisms in 2015 triggered the rise of attention for neural networks. Based on the proposed attention mechanisms, significant research has been conducted in a variety of tasks. In order to imagine the attention idea in neural networks better, two visual examples are shown in Fig. \ref{local attentional model}. A neural image caption generation task is seen in the top row that implements an attention mechanism \cite{xu2015}. Then, the second example shows how the attention mechanisms can be used for visual question answering \cite{lu2017a}. Both examples demonstrate how attention mechanisms focus on parts of input images.

\begin{figure}[h!]
\centering
\includegraphics[scale=.6]{stop}\\
\includegraphics[scale=.6]{snowboard}
\caption{Examples of the attention mechanism in visual. \emph{(Top)} Attending to the correct object in neural image caption generation \cite{xu2015}. \emph{(Bottom)} Visualization of original image and question pairs, and co-attention maps namely word-level, phrase-level and question-level, respectively \cite{lu2017a}.}
\label{local attentional model}
\end{figure}

\section{2015-2016: Attack of the Attention}
\label{sec:3}
During two years from 2015, the attention mechanisms were used for different tasks, and novel neural network architectures were presented applying these mechanisms. After the \emph{memory networks} \cite{weston2014} that require a supervision signal instructing them how to use their memory cells, the introduction of the \emph{neural Turing machine} \cite{graves2014} allows end-to-end training without this supervision signal, via the use of a content-based soft attention mechanism \cite{goodfellow2016}. Then, \emph{end-to-end memory network} \cite{sukhbaatar2015} that is a form of \emph{memory network} based on a recurrent attention mechanism is proposed.

In these years, an attention mechanism called  \emph{self-attention}, sometimes called \emph{intra-attention}, was successfully implemented within a neural network architecture namely \emph{Long Short-Term Memory-Networks (LSTMN)} \cite{cheng2016}. It modifies the standard LSTM structure by replacing the memory cell with a memory network \cite{weston2014}. This is because memory networks have a set of key vectors and a set of value vectors, whereas LSTMs maintain a hidden vector and a memory vector \cite{cheng2016}. In contrast to attention idea in \cite{bahdanau2015}, memory and attention are added \emph{within} a sequence encoder in LSTMN. In order to compute a representation of a sequence, self-attention is described as relating different positions of it \cite{vaswani2017}. One of the first approaches of self-attention is applied for natural language inference \cite{parikh2016}.

Many attention-based models have been proposed for neural image captioning \cite{you2016}, abstractive sentence summarization \cite{rush2015}, speech recognition \cite{yu2016,chorowski2015}, automatic video captioning \cite{zanfir2016}, neural machine translation \cite{cheng2016a}, and recognizing textual entailment \cite{rocktaschel2016}. Different attention-based models perform visual question answering \cite{zhu2016,chen2015,xu2016}. An attention-based CNN is presented for modeling sentence pairs \cite{yin2016}. A recurrent soft attention based model learns to focus selectively on parts of the video frames and classifies videos \cite{sharma2016}.

On the other side, several neural network architectures have been presented in a variety of tasks. For instance, \emph{Stacked Attention Network (SAN)} is described for image question answering \cite{yang2016}. {\em Deep Attention Recurrent Q-Network (DARQN)} integrates soft and hard attention mechanisms into the structure of Deep Q-Network (DQN) \cite{sorokin2015}. {\em Wake-Sleep Recurrent Attention Model (WS-RAM)} speeds up the training time for image classification and caption generation tasks \cite{ba2015}. {\em alignDRAW} model, an extension of the \emph{Deep Recurrent Attention Writer (DRAW)} \cite{gregor2015}, is a generative model of images from captions using a soft attention mechanism \cite{mansimov2016}. \emph{Generative Adversarial What-Where Network (GAWWN)} synthesizes images given instructions describing what content to draw in which location \cite{reed2016}.

\section{The Transformer: Return of the Attention}
\label{sec:4}
After the proposed attention mechanisms in 2015, researchers published studies that mostly modifying or implementing them to different tasks. However, in 2017, a novel neural network architecture, namely the \emph{Transformer}, based entirely on \emph{self-attention} was presented \cite{vaswani2017}. The Transformer achieved great results on two machine translation tasks in addition to English constituency parsing. The most impressive point about this architecture is that it contains neither recurrence nor convolution. The Transformer performs well by replacing the conventional recurrent layers in encoder-decoder architecture used for NMT with self-attention.

The Transformer is composed of encoder-decoder stacks each of which has six identical layers within itself. In Fig. \ref{transformer}, one encoder-decoder stack is shown to illustrate the model \cite{vaswani2017}. Each stack includes only attention mechanisms and feedforward neural networks. As this architecture does not include any recurrent or convolutional layer, information about the relative or absolute positions in the input sequence is given at the beginning of both encoder and decoder using {\emph{positional encodings}.

\begin{figure}[h!]
\centering
\includegraphics[scale=.24]{transformer}
\caption{The Transformer architecture and the attention mechanisms it uses in detail \cite{vaswani2017}. \emph{(Left)} The Transformer with one encoder-decoder stack. \emph{(Center)} Multi-head attention. \emph{(Right)} Scaled dot-product attention.}
\label{transformer}
\end{figure}

The calculations of self-attention are slightly different from the mechanisms described so far in this paper. It uses three vectors namely {\emph{query}}, {\emph{key}} and {\emph{value}} for each word. These vectors are computed by multiplying the input with weight matrices {\emph{$W_q$}}, {\emph{$W_k$}} and {\emph{$W_v$}} which are learned during training. In general, each value is weighted by a function of the query with the corresponding key. The output is computed as a weighted sum of the values. Based on this idea, two attention mechanisms are proposed: In the first one, called \emph{scaled dot-product attention}, the dot products of the query with all keys are computed as given in the right side of Fig. \ref{transformer}. Each result is divided to the square root of the dimension of the keys to have more stable gradients. They pass into the softmax function, thus the weights for the values are obtained. Finally each softmax score is multiplied with the value as given in Eq. (\ref{selfatt}). The authors propose computing the attention on a set of queries simultaneously by taking queries and keys of dimension $d_k$, and values of dimension $d_v$ as inputs. The keys, queries and values are packed together into matrices \emph{K}, \emph{Q} and \emph{V}. Finally, the output matrix is obtained as follows \cite{vaswani2017}:

\begin{equation}
Attention(Q,K,V) = softmax(\frac{QK^T}{\sqrt{d_k}})V
\label{selfatt}
\end{equation}

This calculation is performed by every word against the other words. This leads to having {\emph{values}} of each word relative to each other. For instance, if the word $x_2$ is not relevant for the word $x_1$, then the softmax score gives low probability scores. As a result, the corresponding value is decreased. This leads to an increase in the value of relevant words, and those of others decrease. In the end, every word obtains a new value for itself.

As seen from Fig. \ref{transformer}, the Transformer model does not directly use scaled dot-product attention. But the attention mechanism it uses is based on these calculations. The second mechanism proposed, called the \emph{multi-head attention}, linearly projects the queries, keys and values {\em h} times with different, learned linear projections to $d_q$, $d_k$ and $d_v$ dimensions, respectively \cite{vaswani2017}. The attention function is performed in parallel on each of these projected versions of queries, keys and values, i.e., {\emph{heads}}. By this way, $d_v$-dimensional output values are obtained. In order to get the final values, they are concatenated and projected one last time as shown in the center of Fig. \ref{transformer}. By this way, the self-attention is calculated multiple times using different sets of query, key and value vectors. Thus, the model can jointly attend to information at different positions \cite{vaswani2017}:

\begin{eqnarray}
MultiHead(Q,K,V) = Concat(head_1,...,head_h)W^O  \\
\nonumber    where \; head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)
\end{eqnarray}

In the decoder part of the Transformer, {\emph{masked multi-head attention} is applied first to ensure that only previous word embeddings are used when trying to predict the next word in the sentence. Therefore, the embeddings that shouldn't be seen by the decoder are masked by multiplying with zero.

An interesting study examines the contribution made by individual attention heads in the encoder \cite{voita2019}. Also, there is an evaluation of the effects of self-attention on gradient propagation in recurrent networks \cite{kerg2020}. For a deeper analysis of multi-head self-attention mechanism from a theoretical perspective see \cite{cordonnier2020}.

Self-attention has been used successfully in a variety of tasks including sentence embedding \cite{lin2017} and abstractive summarization \cite{paulus2018}. It is shown that self-attention can lead to improvements to discriminative constituency parser \cite{kitaev2018}, and speech recognition as well \cite{povey2018,vyas2020}. Also, the \emph{listen-attend-spell} model \cite{chan2016} has been improved with the self-attention for acoustic modeling \cite{sperber2018}.

As soon as these self-attention mechanisms were proposed, they have been incorporated with deep neural networks for a wide range of tasks. For instance, a deep learning model learned a number of large-scale tasks from multiple domains with the aid of self-attention mechanism \cite{kaiser2017}. Novel self-attention neural models are proposed for cross-target stance classification \cite{xu2018a} and NMT \cite{maruf2019}. Another study points out that a fully self-attentional model can reach competitive predictive performance on ImageNet classification and COCO object detection tasks \cite{ramachandran2019}. Besides, developing novel attention mechanisms has been carried out such as \emph{area attention}, a novel mechanism that can be used along multi-head attention \cite{li2019}. It attends to areas in the memory by defining the key of an area as the mean vector of the key of each item, and defining the value as the sum of all value vectors in the area.

When a novel mechanism is proposed, it is inevitable to incorporate it into the GAN framework \cite{goodfellow2014}. \emph{Self-Attention Generative Adversarial Networks (SAGANs)} \cite{zhang2019} introduce a self-attention mechanism into convolutional GANs. Different from the traditional convolutional GANs, SAGAN generates high-resolution details using cues from all feature locations. Similarly, \emph{Attentional Generative Adversarial Network (AttnGAN)} is presented for text to image generation \cite{xu2018}. On the other side, a machine reading and question answering architecture called {\em QANet} \cite{yu2018} is proposed without any recurrent networks. It uses self-attention to learn the global interaction between each pair of words whereas convolution captures the local structure of the text. In another study, \emph{Gated Attention Network (GaAN)} controls the importance of each attention head's output by introducing gates \cite{zhang2018a}. Another interesting study introduces \emph{attentive group convolutions} with a generalization of visual self-attention \cite{romero2020}. A deep transformer model is implemented for language modeling over long sequences \cite{alrfou2019}.

\begin{table*}
{\setlength{\tabcolsep}{3pt}
\caption{Summary of Notation}
\begin{center}
\vspace{-2mm}
\begin{tabular}{rccc}
\hline\hline
Symbol & Definition \\
\hline
a   & annotation   \\
c   & context vector  \\
$\alpha$  & weight  \\
e  & energy  \\
f & feedforward neural network  \\
h & hidden state  \\
$\phi$ & hard (stochastic) / soft (deterministic) attention  \\
s & location variable  \\
p & source position \\
$K$, $Q$, $V$ & keys, queries and values matrices, respectively \\
$W_q$, $W_k$, $W_v$ & weight matrices for queries, keys and values, respectively \\
\hline
\end{tabular}
\vspace{-2mm}
\end{center}
\label{table}}
\end{table*}

\subsection{Self-attention variants}

In recent years, self-attention has become an important research direction within the deep learning community. Self-attention idea has been examined in different aspects. For example, self-attention is handled in a multi-instance learning framework \cite{du2018}. The idea of \emph{Sparse Adaptive Connection (SAC)} is presented for accelerating and structuring self-attention \cite{li2020a}. The research on improving self-attention continues as well \cite{yang2019,yang2018a,bello2019}. Besides, based on the self-attention mechanisms proposed in the Transformer, important studies that modify the self-attention have been presented. Some of the most recent and prominent studies are summarized below.

\paragraph{\textbf{Relation-aware self-attention}}

It extends the self-attention mechanism by regarding representations of the relative positions, or distances between sequence elements \cite{shaw2018}. Thus, it can consider the pairwise relationships between input elements. This type of attention mechanism defines vectors to represent the edge between two inputs. It provides learning two distinct edge representations that can be shared across attention heads without requiring additional linear transformations.

\paragraph{\textbf{Directional self-attention (DiSA)}} A novel neural network architecture for learning sentence embedding named \emph{Directional Self-Attention Network (DiSAN)} \cite{shen2017} uses \emph{directional} self-attention followed by a \emph{multi-dimensional} attention mechanism. Instead of computing a single importance score for each word based on the word embedding,  multi-dimensional attention computes a feature-wise score vector for each token. To extend this mechanism to the self-attention, two variants are presented: The first one, called \emph{multi-dimensional `token2token' self-attention} generates context-aware coding for each element. The second one, called \emph{multi-dimensional `source2token' self-attention} compresses the sequence into a vector \cite{shen2017}. On the other side, directional self-attention produces context-aware representations with temporal information encoded by using positional masks. By this way, directional information is encoded. First, the input sequence is transformed to a sequence of hidden states by a fully connected layer. Then, multi-dimensional token2token self-attention is applied to these hidden states. Hence, context-aware vector representations are generated for all elements from the input sequence.

\paragraph{\textbf{Reinforced self-attention (ReSA)}}

A sentence-encoding model named \emph{Reinforced Self-Attention Network (ReSAN)} uses \emph{reinforced self-attention (ReSA)} that integrates soft and hard attention mechanisms into a single model. ReSA selects a subset of head tokens, and relates each head token to a small subset of dependent tokens to generate their context-aware representations \cite{shen2018}. For this purpose, a novel hard attention mechanism called \emph{reinforced sequence sampling (RSS)}, which selects tokens from an input sequence in parallel and trained via policy gradient, is proposed. Given an input sequence, RSS generates an equal-length sequence of binary random variables that indicates both the selected and discarded ones. On the other side, the soft attention provides reward signals back for training the hard attention. The proposed RSS provides a sparse mask to self-attention. ReSA uses two RSS modules to extract the sparse dependencies between each pair of selected tokens.

\paragraph{\textbf{Outer product attention (OPA)}}

\emph{Self-Attentive Associative Memory (SAM)} is a novel operator based upon \emph{outer product attention (OPA)} \cite{le2020}. This attention mechanism is an extension of dot-product attention \cite{vaswani2017}. OPA differs using element-wise multiplication, outer product, and \emph{tanh} function instead of \emph{softmax}.

\paragraph{\textbf{Bidirectional block self-attention (Bi-BloSA)}}

Another mechanism, \emph{bidirectional block self-attention (Bi-BloSA)} which is simply a \emph{masked block self-attention (mBloSA)} with forward and backward masks to encode the temporal order information is presented \cite{shen2018a}. Here, mBloSA is composed of three parts from its bottom to top namely \emph{intra-block self-attention}, \emph{inter-block self-attention} and \emph{the context fusion}. It splits a sequence into several length-equal blocks, and applies an intra-block self-attention to each block independently. Then, inter-block self-attention processes the outputs for all blocks. This stacked self-attention model results a reduction in the amount of memory compared to a single one applied to the whole sequence. Finally, a feature fusion gate combines the outputs of intra-block and inter-block self-attention with the original input, to produce the final context-aware representations of all tokens.

\paragraph{\textbf{Fixed multi-head attention}}

The \emph{fixed multi-head attention} proposes fixing the head size of the Transformer in the aim of improving the representation power \cite{bhojanapalli2020}. This study emphasizes its importance by setting the head size of attention units to input sequence length.

\paragraph{\textbf{Sparse sinkhorn attention}}

It is based on the idea of differentiable sorting of internal representations \emph{within} the self-attention module \cite{tay2020}. Instead of allowing tokens to only attend to tokens within the same block, it operates on block sorted sequences. Each token attends to tokens in the {\em sorted} block. Thus, tokens that may be far apart in the unsorted sequence can be considered. Additionally, a variant of this mechanism named \emph{SortCut sinkhorn attention} applies a post-sorting truncation of the input sequence.

\paragraph{\textbf{Adaptive attention span}}

\emph{Adaptive attention span} is proposed as an alternative to self-attention \cite{sukhbaatar2019}. It learns the attention span of each head independently. To this end, a masking function inspired by \cite{jernite2017} is used to control the attention span for each head. The purpose of this novel mechanism is to reduce the computational burden of the Transformer.
Additionally, \emph{dynamic attention span} approach is presented to dynamically change the attention span based on the current input as an extension \cite{luong2015,shu2017}.

\subsection{Transformer variants}

Different from developing novel self-attention mechanisms, several studies have been published in the aim of improving the performance of the Transformer. These studies mostly modify the model architecture. For instance, an additional recurrence encoder is preferred to model recurrence for Transformer directly \cite{hao2019}. In another study, a new weight initialization scheme is applied to improve Transformer optimization \cite{huang2020}. A novel positional encoding scheme is used to extend the Transformer to tree-structured data \cite{shiv2019}. Investigating model size by handling Transformer width and depth for efficient training is also an active research area \cite{li2020}. Transformer is used in reinforcement learning settings \cite{hoshen2017,hu2021,parisotto2021} and for time series forecasting in adversarial training setting \cite{wu2020a}.

Besides, many Transformer variants have been presented in the recent past. \emph{COMmonsEnse Transformer (COMET)} is introduced for automatic construction of commonsense knowledge bases \cite{bosselut2019}. \emph{Evolved Transformer} applies neural architecture search for a better Transformer model \cite{so2019}. \emph{Transformer Autoencoder} is a sequential autoencoder for conditional music generation \cite{choi2020}. \emph{CrossTransformer} takes a small number of labeled images and an unlabeled query, and computes distances between spatially-corresponding features to infer class membership \cite{doersch2020}.  \emph{DEtection TRansformer (DETR)} is a new design for object detection systems \cite{carion2020}, and \emph{Deformable DETR} is an improved version that achieves better performance in less time \cite{zhu2021}. \emph{FLOw-bAsed TransformER (FLOATER)} emphasizes the importance of position encoding in the Transformer, and models the position information via a continuous dynamical model \cite{liu2020}.  \emph{Disentangled Context (DisCo) Transformer} simultaneously generates all tokens given different contexts by predicting every word in a sentence conditioned on an arbitrary subset of the rest of the words \cite{kasai2020}. \emph{Generative Adversarial Transformer (GANsformer)} is presented for visual generative modeling \cite{hudson2021}.

Recent work has demonstrated significant performance on NLP tasks. In \emph{OpenAI GPT}, there is a left-to-right architecture, where every token can only attend to previous tokens in the self-attention layers of the Transformer \cite{radford2018}. \emph{GPT-2} \cite{radford2019} and \emph{GPT-3} \cite{brown2020} models have improved the progress. In addition to these variants, some prominent Transformer-based models are summarized below.

\paragraph{\textbf{Universal Transformer}}

A generalization of the Transformer model named the \emph{Universal Transformer} \cite{dehghani2018} iteratively computes representations $H^t$ at step {\em t} for all positions in the sequence in parallel. To this end, it uses the scaled dot-product attention in Eq. (\ref{selfatt}) where {\em d} is the number of columns of Q, K and V. In the Universal Transformer, the multi-head self-attention with {\em k} heads is used. The representations $H^t$ is mapped to queries, keys and values with affine projections using learned parameter matrices $W^Q \in \Re^{d\times d/k}$, $W^K \in \Re^{d\times d/k}$, $W^V \in \Re^{d\times d/k}$ and $W^O \in \Re^{d\times d}$ \cite{dehghani2018}:

\begin{eqnarray}
MultiHead(H^t) = Concat(head_1,...,head_k)W^O  \\
\nonumber where \; head_i = Attention(H^tW_i^Q, H^tW_i^K, H^tW_i^V)
\label{multiheadselfuniversaltransformer}
\end{eqnarray}

\paragraph{\textbf{Image Transformer}}

\emph{Image Transformer} \cite{parmar2018} demonstrates that self-attention based models can also be well-suited for images instead of text. This Transformer type restricts the self-attention mechanism to attend to local neighborhoods. Thus, the size of images that the model can process is increased. Its larger receptive fields allow the Image Transformer to significantly improve the model performance on image generation as well as image super-resolution.

\paragraph{\textbf{Transformer-XL}}

This study aims to improve the fixed-length context of the Transformer \cite{vaswani2017} for language modeling. \emph{Transformer-XL} \cite{dai2019} makes modeling very long-term dependency possible by reusing the hidden states obtained in previous segments. Hence, information can be propagated through the recurrent connections. In order to reuse the hidden states without causing temporal confusion, Transformer-XL uses relative positional encodings. Based on this architecture, a modified version named \emph{the Gated Transformer-XL (GTrXL)} is presented in the reinforcement learning setting \cite{parisotto2020}.

\paragraph{\textbf{Tensorized Transformer}}

\emph{Tensorized Transformer} \cite{ma2019} compresses the multi-head attention in Transformer. To this end, it uses a novel self-attention model \emph{multi-linear attention} with Block-Term Tensor Decomposition (BTD) \cite{lathauwer2008}. It builds a \emph{single-block attention} based on the Tucker decomposition \cite{tucker1966}. Then, it uses a multi-linear attention constructed by a BTD to compress the multi-head attention mechanism. In Tensorized Transformer, the factor matrices are shared across multiple blocks.

\paragraph{\textbf{BERT}}

The \textbf{B}idirectional \textbf{E}ncoder \textbf{R}epresentations from \textbf{T}ransformers \emph{(BERT)}
aims to pre-train deep bidirectional representations from unlabeled text \cite{devlin2019}. BERT uses a multilayer bidirectional Transformer as the encoder. Besides, inspired by the Cloze task \cite{taylor1953}, it has a \emph{masked language model} pre-training objective. BERT randomly masks some of the tokens from the input, and predicts the original vocabulary id of the masked word based only on its context. This model can pre-train a deep bidirectional Transformer. In all layers, the pre-training is carried out by jointly conditioning on both left and right context. BERT differs from the left-to-right language model pre-training from this aspect.

Recently, BERT model has been examined in detail. For instance, the behaviour of attention heads are analysed \cite{clark2019}. Various methods have been investigated for compressing \cite{sun2019,wang2020a}, pruning \cite{mccarley2020}, and quantization \cite{zafrir2019}. Also, BERT model has been considered for different tasks such as coreference resolution \cite{joshi2019b}. A novel method is proposed in order to accelerate BERT training \cite{gong2019}.

Furthermore, various BERT variants have been presented. \emph{ALBERT} aims to increase the training speed of BERT, and presents two parameter reduction techniques \cite{lan2020}. Similarly, \emph{PoWER-BERT} \cite{goyal2020} is developed to improve the inference time of BERT. This scheme is also used to accelerate ALBERT. Also, \emph{TinyBERT} is proposed to accelerate inference and reduce model size while maintaining accuracy \cite{jiao2019}. In order to obtain better representations, \emph{SpanBERT} is proposed as a pre-training method \cite{joshi2020}. As a robustly optimized BERT approach, \emph{RoBERTa} shows that BERT was significantly undertrained \cite{liu2019}. Also, \emph{DeBERTa} improves RoBERTa using the disentangled attention mechanism \cite{he2021}. On the other side, \emph{DistilBERT} shows that it is possible to reach similar performances using much smaller language models pre-trained with knowledge distillation \cite{sanh2019}. \emph{StructBERT} proposes two novel linearization strategies \cite{wang2020}. \emph{Q-BERT} is introduced for quantizing BERT models \cite{shen2020}, \emph{BioBERT} is for biomedical text mining \cite{lee2020}, and
\emph{RareBERT} is for rare disease diagnosis \cite{prakash2021}.

Since 2017 when the Transformer was presented, research directions have generally focused on novel self-attention mechanisms, adapting the Transformer for various tasks, or making them more understandable. In one of the most recent studies, NLP becomes possible in the mobile setting with {\emph{Lite Transformer}. It applies {\emph{long-short range attention} where some heads specialize in the local context modeling while the others specialize in the long-distance relationship modeling \cite{wu2020}. A deep and light-weight Transformer \emph{DeLighT} \cite{mehta2021} and a hypernetwork-based model namely \emph{HyperGrid Transformers} \cite{tay2021} perform with fewer parameters. \emph{Graph Transformer Network} is introduced for learning node representations on heterogeneous graphs    \cite{yun2019} and different applications are performed for molecular data \cite{rong2020} or textual graph representation \cite{yang2021}. Also, {\emph{Transformer-XH} applies {\emph{eXtra Hop attention} for structured text data \cite{zhao2020}. \emph{AttentionXML} is a tree-based model for extreme multi-label text classification \cite{you2019}. Besides, attention mechanism is handled in a Bayesian framework \cite{fan2020}. For a better understanding of Transformers, an identifiability analysis of self-attention weights is conducted in addition to presenting {\emph{effective attention} to improve explanatory interpretations \cite{brunner2020}. Lastly, \emph{Vision Transformer (ViT)} processes an image using a standard Transformer encoder as used in NLP by interpreting it as a sequence of patches, and performs well on image classification tasks \cite{dosovitskiy2021}.

\subsection{What about complexity?}

All these aforementioned studies undoubtedly demonstrate significant success. But success not make one great. The Transformer also brings a very high computational complexity and memory cost. The necessity of storing attention matrix to compute the gradients with respect to queries, keys and values causes a non-negligible quadratic computation and memory requirements. Training the Transformer is a slow process for very long sequences because of its quadratic complexity. There is also time complexity which is quadratic with respect to the sequence length. In order to improve the Transformer in this respect, recent studies have been conducted to improve this issue. One of them is \emph{Linear Transformer} which expresses the self-attention as a linear dot-product of kernel feature maps \cite{katharopoulos2020}. Linear Transformer reduces both memory and time complexity by changing the self-attention from the softmax function in Eq. (\ref{selfatt}) to a feature map based dot-product attention. Its performance is competitive with the vanilla Transformer architecture on image generation and automatic speech recognition tasks while being faster during inference. On the other side, \emph{FMMformers} which use the idea of the \emph{fast multipole method (FMM)} \cite{leslie1987} outperform the linear Transformer by decomposing the attention matrix into near-field and far-field attention with linear time and memory complexity \cite{nguyen2021}.

Another suggestion made in response to the Transformer's quadratic nature is The \emph{Reformer} that replaces dot-product attention by one that uses \emph{locality-sensitive hashing} \cite{kitaev2020}. It reduces the complexity but one limitation of the Reformer is its requirement for the queries and keys to be identical. \emph{Set Transformer} aims to reduce computation time of self-attention from quadratic to linear by using an attention mechanism based on sparse Gaussian process literature \cite{lee2019}. \emph{Routing Transformer} aims to reduce the overall complexity of attention by learning dynamic sparse attention patterns by using \emph{routing attention with clustering} \cite{roy2020}. It applies k-means clustering to model sparse attention matrices. At first, queries and keys are assigned to clusters. The attention scheme is determined by considering only queries and keys from the same cluster. Thus, queries are routed to keys belonging to the same cluster \cite{roy2020}.

\emph{Sparse Transformer} introduces sparse factorizations of the attention matrix by using \emph{factorized self-attention}, and avoids the quadratic growth of computational burden \cite{child2019}. It also shows the possibility of modeling sequences of length one million or more by using self-attention in theory. In the Transformer, all the attention heads with the softmax attention assign a non-zero weight to all context words. \emph{Adaptively Sparse Transformer} replaces softmax with $\alpha$-entmax which is a differentiable generalization of softmax allowing low-scoring words to receive precisely zero weight \cite{correia2019}. By means of context-dependent sparsity patterns, the attention heads become flexible in the Adaptively Sparse Transformer. \emph{Random feature attention} approximates softmax attention with random feature methods \cite{peng2021}. \emph{Skyformer} replaces softmax with a Gaussian kernel and adapts Nyström method \cite{chen2021}.  A sparse attention mechanism named \emph{BIGBIRD} aims to reduce the quadratic dependency of Transformer-based models to linear \cite{zaheer2020}. Different from the similar studies, BIGBIRD performs well for genomics data alongside NLP tasks such as question answering.

\emph{Music Transformer} \cite{huang2019} shows that self-attention can also be useful for modeling music. This study emphasizes the infeasibility of the relative position representations introduced by \cite{shaw2018} for long sequences because of the quadratic intermediate relative information in the sequence length. Therefore, this study presents an extended version of relative attention named \emph{relative local attention} that improves the relative attention for longer musical compositions by reducing its intermediate memory requirement to linear in the sequence length. A softmax-free Transformer (\emph{SOFT}) is presented to improve the computational efficiency of ViT. It uses Gaussian kernel function instead of the dot-product similarity \cite{lu2021}.

Additionally, various approaches have been presented in {\emph{Hierarchical Visual Transformer} \cite{pan2021}, \emph{Long-Short Transformer (Transformer-LS)} \cite{zhu2021a}, \emph{Perceiver} \cite{jaegle2021}, and  {\emph{Performer}} \cite{choromanski2021}. Image Transformer based on the cross-covariance matrix between keys and queries is applied \cite{elnouby2021}, and a new vision Transformer is proposed \cite{yu2021}. Furthermore, a Bernoulli sampling attention mechanism decreases the quadratic complexity to linear \cite{zeng2021}. A novel linearized attention mechanism performs well on object detection, instance segmentation, and stereo depth estimation \cite{shen2021}. A study shows that kernelized attention with relative positional encoding can be calculated using Fast Fourier Transform and it leads to get rid of the quadratic complexity for long sequences \cite{luo2021}. A linear unified nested attention mechanism namely \emph{Luna} uses two nested attention functions to approximate the softmax attention in Transformer to achieve linear time and space complexity \cite{ma2021}.

\section{Concluding Remarks: A New Hope}
\label{conc}
Inspired by the human visual system, the attention mechanisms in neural networks have been developing for a long time. In this study, we examine this duration beginning with its roots up to the present time. Some mechanisms have been modified, or novel mechanisms have emerged in this period. Today, this journey has reached a very important stage. The idea of incorporating attention mechanisms into deep neural networks has led to state-of-the-art results for a large variety of tasks. Self-attention mechanisms and \emph{GPT-n} family models have become a new hope for more advanced models. These promising progress bring the questions whether the attention could help further development, replace the popular neural network layers, or could be a better idea than the existing attention mechanisms? It is still an active research area and much to learn we still have, but it is obvious that more powerful systems are awaiting when neural networks and attention mechanisms join forces.

 \section*{Conflict of interest}
The author declares that she has no conflict of interest.

\bibliographystyle{spphys}


Marching Band – This group is made up of all the students in the program including Brass, Woodwinds, Drumline, and Pageantry. The Marching Band performs at various events during the fall season. These events include home football games, school rallies, parades, and field show competitions during the first semester.
Pageantry – Our Pageantry is made up of our Tall Flags, Drill Team, Shield Line, and Winterguard. These auxiliary units provide the color, life, and movement to the music performed by the band. Our pageantry teams compete with the marching band during the first semester and then individually during the second semester.
Concert Band – The various concert bands focus on learning and performing symphonic works from various time periods. This educates music students on content area state standards that are evaluated at by Festival Judges. The concert band, symphonic band, and wind ensemble perform in our very own Cook Auditorium during second semester.
Colonist Jazz Band – Is an auditioned group that is made up of saxophones, trumpets, trombones, and rhythm section. This group performs a variety of Jazz styles and showcases their talent at Jazz Festivals during second semester.


text:
Lara Karaian (Carleton University)
Pornography as a "Public Health" Crisis: Censoring Sexual Speech Through the Back Door
Pornography has once again been thrust onto the public stage. This time, however, it's been (re)framed as a "public health" issue (as opposed to a women's equality issue or a source of criminal harm). I've been researching sexual speech for almost as long as I've been consuming it, which is to say a long time. It therefore comes as no surprise that we're in the midst of yet another attempt to censor it and to surveille its consumers.
In March of 2016, Utah's House of Representatives passed a resolution declaring porn a "public health hazard leading to a broad spectrum of individual and public health impacts and societal harms". Numerous states have since followed suit. The Republican Party, in its 2016 party platform, has described pornography as "a public health crisis that is destroying the lives of millions" and encouraged states to "continue to fight this public menace." Even former playboy bunny (14 times over) Pamela Anderson has co-authored an op-ed in the Washington Post where she has called pornography "a public hazard of unprecedented seriousness" and "a boring wasteful and dead-end outlet for people too lazy to reap the ample rewards of healthy sexuality."
Canada (along with the UK, Australia and New Zealand) has also witnessed a similar public debate. On March 8th, 2016 Canadian Conservative MP Arnold Viersen introduced private member's motion--Motion M-47--instructing the Standing Committee on Health to "examine the public health effects of the ease of access and viewing of online violent and degrading sexually explicit material on children, women and men" and requiring the Committee to report its findings to the House no later than July 2017. According to Vierson a motion of this nature was necessary in light of "recent national incidents of sexual violence in the last few years [Amanda Todd, Rehtaea Parsons, and the Dalhousie dentistry students]" which lead him to wonder "where these individuals get the idea that this behavior involving sexual violence is acceptable." While Vierson seems to have presupposed a causal link between porn and rape despite much evidence to the contrary, MPs from all five parties nevertheless passed M-47 unanimously and, following months of hearings, the Health Committee released its final report on June 9, 2017.
My commentary is limited to a few reflections on the censorship and surveillance issues arising from the recent debate, both nationally and abroad. There is a small body of scholarly evidence that has found correlation between sexually explicit imagery and negative consequences for individuals and for families. The question is whether this research is methodologically sound; whether it is representative of all the possible outcomes of exposure to porn; and whether research demonstrating alternative consequences also exists and is persuasive. My twenty plus years of experience researching in this field leads me to conclude that while some of this research may be valid, much of it is not: many studies' methodologies are flawed or biased; the evidence about porn's harms is regularly over-stated; and a large body of case law and evidence has demonstrated the potential benefits of creating and consuming sexually explicit material (see this brief for more on the issue of evidence).
Reframing porn as a "public heath" issue is not an insignificant development. The meaning of "public health" has gradually expanded over the past few decades. Where it was once used to refer to the securing of public health benefits—clear water, improved sanitation, disease control—it is now often used in reference to opioid epidemics, gun violence and obesity. To frame pornography as a "public health" issue is to elevate it to the status of extremely influential and harmful speech requiring exceptional regulation. (At the same time, we are not seeing a call for a ban on sexual or violent representations in mainstream or video games, for example). Referring to porn as a public health issue also suggests that it requires a more structural and collective response. The entire thrust of contemporary anti-porn advocacy is that it is not up to individuals or families to regulate themselves and their offspring but rather a state-led response is necessary. Analogizing the threat of porn to the threat of cigarettes and drugs, contemporary anti-porn advocates thus implore the state to step in and control access in much the same way as tobacco is currently regulated.
So, what are anti-porn advocates demanding of the state and why is their cause for concern? To begin with, they are calling for the implementation of "meaningful age verification", a proposal that our Standing Committee on Health did not recommend in its final report (and has therefore been reiterated by the anti-porn folks in a dissenting report representing the views of the Conservative Members of the Committee). Meaningful age verification—as is used with the gambling industry—requires a user to provide a credit card or third-party verification system to prove that you are 18 before you can access porn. The UK has already legislated this in its Digital Economy Act which came into force in April 2017. In the UK, the age verification regulator requires internet service providers to block pornographic websites that do not offer age verification therefore de facto censoring huge swaths of legal sexual speech. This age verification provision has caused concern regarding the privacy of collected user data and the possible ineffectiveness of a method focused on restricting payments. Others have expressed concern that it is likely to lead to a database of viewing habits being held by companies or by the Government, which experts have warned could easily be hacked and exposed.
Anti-porn advocates are also asking for the state to label all pornography (not just "violent" porn) as harmful because of its purported addictive qualities. While porn may be addictive to some people, psychological associations have refused to include sex and porn addiction alongside drug addiction in clinical guidelines because the concept of addiction is precise and is meant to be used in ways that accurately reflects a widespread occurrence. This demand for a "harmful warning" begs questions about why sexual speech (that is not even deemed obscene) is subject to more government regulation than other speech acts and other sites where violence, sexuality, and racist attitudes are expressed. Moreover, those who tout the overwhelming "harmfulness" of porn do not appear to factor the pleasures and benefits of sexual speech into their calculation. Porn is rarely if ever presumed to have an intrinsic value (like artistic or scientific speech). In fact, it is rarely acknowledged as valid speech at all. The description of porn as predominantly or merely arousing emotions, desires, and thoughts within us that are allegedly wholly negative, uncontrollable, and implicitly less valuable than other speech that influences and incites is not only unsupported it also exceptionalizes sexual speech and is an indication of the sexual prurience that informs this position.
Finally, the "porn as health menace" folks are demanding that the government force internet service providers to develop opt-in filters whereby all pornography is de facto restricted unless you opt-in to accessing it. Again, a key issue here is the privacy implications and general aversion to the development of list of people who choose to see porn online-what at least one person referred to as being deemed a "Pervert registry" potentially searchable by employers, lending institutions, and insurance companies. Even if our government chooses not to adopt this opt-in model of censorship, attempts to "merely" block content that is deemed obscene by governmental committees would also invite a different set of censorship issues. For example, in the UK--the jurisdiction cited most favourably by the anti-porn folks--the Digital Economy Act also forces internet service providers to block sites hosting content that would not be certified for commercial DVD sale by the British Board of Film Classification (BBFC). This has meant that "nonconventional" content, such as "[p]ictures and videos that show spanking, whipping or caning that leaves marks, and sex acts involving urination, female ejaculation or menstruation as well as sex in public are likely to be caught by the ban." In other words, content that is widely associated with feminist, queer, kink, but also mainstream communities internationally. Notably, in Canada we've already successfully (somewhat) fought censorship of this nature—instigated by border security agencies who consistently confiscated queer and kink porn—at the Supreme Court of Canada in Little Sisters Book and Art Emporium v. Canada (Minister of Justice) [2000].
So, where does this leave us? The Government has 120 days from the tabling of the June 9th Report to respond. I'll be watching. Maybe you will be too?
*cross posted with Ryerson University's Centre for Free Expression blog
surveillance of sex
public health and porn
Lara Karaian
porn and harm
digital dangers of porn
opt in filters

redpajama_set_name: RedPajamaCommonCrawl


text:
Higher US Treasury Yields — Repricing Term Premium does not Harken Back to the "Taper Tantrum"
Published on March 23, 2021 by Manjunath Deshpande and Senthil Kumar Sekar
Long-term US Treasury yields have been rising since late 2020, as the extraordinary monetary and fiscal stimulus have made investors jittery about a potential inflationary shock. Although real yields are edging higher, they are still in negative territory.
The fast-paced vaccination programme, coupled with strong economic recovery and government stimulus package, is expected to prop up consumption and investment. A spike in growth fuel consumption adds inflationary pressure. Hence, investors believe Fed to intervene earlier than anticipated in response to rising yield and inflation.
Cyclical sectors of the economy and Asia remain attractive opportunities in the fixed income market. Furthermore, pockets of opportunity remain in the broader investment grade and HY fixed income market.
Long-term US Treasury yields have increased ca 60bps in 2021, indicating investors are pricing in an optimistic global economic outlook plus inflation concerns. However, investors were caught off-guard on 25 February 2021, when US Treasuries saw a sharp sell-off as 10-year Treasury yields rose 22bps to 1.52%. Consequently, the bid-to-cover ratio, measure of demand, fell to a record low of 2.04 times. Primary dealers were forced to buy nearly 40% of the sale, the highest share in the last seven years, as per Jefferies.
Although the sell-off was triggered by weak demand for US Treasuries, the rise in Treasury yields is broadly attributable to:
USD1.9trn fiscal package (including USD1trn direct aid to individuals and USD440bn to businesses)
Successful vaccine rollout (ca 21% of the US population received the first shot of the vaccine and ca 11% population was fully vaccinated as of mid-March)
Strong economic data — Unemployment rate has been consistently declining, from the peak 14.8% in April 2020 to 6.2% in February 2021, Q4 2020 GDP grew 4.1% and personal income and consumer spending increased 10.0% and 2.4%, respectively, in January 2021.
Case for inflation
The 61bps rise in 10-year nominal yield (as of 11 March 2021) explained by 34bps increase in US Treasury (UST) real yield and 27bps due to expected inflation. Break-even inflation (BEI) rate, the spread between nominal UST and real UST, comprises two components: expected inflation plus premium for the uncertainty surrounding future inflation. The graph below shows rising BEI rate, which indicates investors' expectation (BEI rate — 2.2% in February 2021) of inflation is higher than actual inflation (1.7% in February 2021). We believe this is due to, higher household savings (US households currently holds USD1.4trn in excess savings), resulting in a higher demand post vaccination and strong monetary and fiscal support reviving business growth.
Fixed income asset performance
Higher bond yields have varied implications on asset classes. While HY bonds are negatively correlated to rising Treasury yields, IG bonds remains highly correlated to the treasuries. Regardless, the cyclical sectors of the economy look prime to benefit from improving economic sentiments. Furthermore, global asset managers remain neutral to positive on Emerging Market debt, particularly Asia, which has fared better than other economies in rolling out vaccines and restarting the economy.
Back to fundamentals
Rising yield is an indicator of an increase in investors' higher expectation on term premium, which is not an allegory to the taper tantrum of 2013. Overall, we believe after enduring a volatile 2020 and a relatively weak start to 2021, debt investors have several positive things to look forward to in 2021. Valuations, bottom-up fundamentals and top-down factors remain attractive for multiple assets classes in the broader fixed income universe, signalling potential strong returns in 2021.
The volatile market over the past year has forced asset managers to reallocate their portfolios and revaluate their risks. The debt market will likely remain volatile in the near future, and asset managers are switching back to deep fundamental reviews of issuers to understand their risks and opportunities. Acuity Knowledge Partners supports some of the largest asset managers globally, which have leveraged our credit research expertise over the years. Our time-bound scalable solutions have helped asset managers of all scale to develop their proprietary solutions.
Manjunath, Delivery Manager, has over 10 years of experience in fixed income credit research, with a focus on HY and IG issuers. At Acuity Knowledge Partners, he supports a leading European investment bank's HY desk, preparing financial models, tracking the portfolio and updating charts and earnings summaries. Earlier, he supported a sell-side research team of Europe's leading investment bank through a wide range of research on EM issuers. Manjunath holds an MBA (Finance) from Shivaji University, Kolhapur and a Bachelor of Commerce from Karnataka University, Dharwad.
Senthil, Senior Associate, has above 5 years of experience in investment research and has been associated with Acuity Knowledge Partners since June 2018. In his current role, he covers HY issuers for a sell-side desk analyst. Prior to joining Acuity Knowledge Partners, he was the lead analyst at WNS Global Services. Senthil holds a PGPM from ICFAI Business School Bangalore, and Bachelor of Technology (Chemical Engineering) from ACT, Anna University. He is also a CFA Level II candidate.
Pyper Maestro in Pyper Maestro
MONEY IS NOW FREE!!!
Stock Market Free Fall Immediately Ahead!
Guy Sharp
Technological Unemployment: Why Keynes Is More Relevant Than Ever
Estela Souto in New Tech Revolution @sciencespo
How to Expand Research Coverage of China bonds
Original NAFTA Negotiators Take Aim at Trump's Trade Talk
Dark Market Economist
We Are In a Moral Great Depression After America's Ethico-Legal Crash.
Albert Goins
Most unlikely climate actor of 2018 award goes to …
Denise Young 楊 玲 玲


To realize technical solutions that satisfies customer's needs.
To work following our agreed procedures and matching our quality standards.
To seek assistance every time you are in the need, not trying to shine by self-love or obstinacy.
You perceive the overall impacts of your day-to-day choices and know when to ask for validation. You face challenges proactively, not always waiting for others to solve problems for you, but know when to stop and ask for help to avoid wasting time or losing your mind on things bigger than you.
You realize solutions that fulfill business goals, matching our high quality standards, as defined with your team.
You learn new, useful skills, taking advantage of any opportunity, balancing possible negative impacts on your duties.
You help improve internal projects (even if just with bugfixes or documentation) based on your skills, aspirations and training goals.
You propose or help to generate content about technology, methodology or practices for Sparkfabrik's media.
You deliver contributions (bugfixes, documentation, translations) to community projects based on your skills, aspirations and training goals.


Our Appetite Controlled Diet consists of eating 3 meals per day and snacks when hungry. You are able to eat fruits, vegetables, whole-grains (such as bread, cereal, pasta), and a healthful type of protein each day! Our diet is tailored to you, because at your initial visit we go over the diet and determine your specific calorie needs. We help you learn what the best foods are to eat, and how many servings you should have per day. Our diet plan gives you examples which will help you stick the diet without restricting too many foods. Many people in the Minneapolis and St Paul areas say that they don't even feel like they are on a diet when they start our program because the foods they are eating help them feel full longer and there are very few limitations.
5. Pregnant, breast feeding, or trying to get pregnant.
12. On SSRI medication for anxiety/depression.
Is a deposit required to schedule an appointment?
Yes, a deposit of $100 is required of all patients. (payable by Visa or MasterCard credit/debit card only) is required of all patients (If you do not have a credit card account, pre-paid Visa/MasterCards are available at many stores.) The deposit is fully refundable until 7 calendar days prior to the appointed office visit date, non-refundable thereafter for either cancellation or postponement.
The first month of our Appetite Controlled Diet program costs $299. Subsequent months of the Appetite Controlled Diet program cost is $199.


Blog Teams Arsenal No opposition, open goal – how long would Arsenal take to score?
No opposition, open goal – how long would Arsenal take to score?
Everyone has special places where they get their brightest ideas – in the shower, on the crapper, while walking, or maybe while they're trying to sleep.
If there was no opposition team member on the pitch except the goal keeper, how long would it take Arsenal to score a goal?
Would they adapt to the situation and play a long ball from defence to attack to cover distance quickly? Would they flood the box and float in a cross?
Would Arsenal have players running in the box and shooting at the keeper because they had a clear shot?
Or would Arsenal pass the ball around, play one-twos outside the box, pass it around inside the box and try to get the keeper to yawn from boredom before shooting at an empty goal and missing it completely?
It's interesting to take this scenario and then apply it to other clubs – how would Chelsea or Manchester United play this?
Be honest, and the answer will surprise you. Because if you honestly believe that Arsenal would play 'direct' football because they have the opportunity to do so, then tell me why they haven't so far. Not for a lack of opportunity, surely.
Are Manchester City too petty to reject Manchester United's request?
I think they'd spend most of the time taking turns to see who could hit the bar.
Obviously you haven't realised that this season we have played more directly than in most of the last 8. interesting that this has been a bad season. you could argue our old play it round the houses technique has led to our current troubles but i would argue that this season we have lack an 'unlocking' player. i love henry but when he wasn't captain he was happy to be the focus of the team. now he wants others to take responsibility. he was a poor chance for captaincy. or was he remember the bernabeu.
Mike – obviously I haven't.
"Couldn't hit a barn door with a banjo" comes to mind here.
But Chris, I'm sure Arsenal can score. It's just a question of how long they would take to score.
And I ask this, quite seriously now, because it's a question of adaptibility. Arsenal this season have been inflexible and haven't adapted / responded to difficult situations.
Chelsea would punt it up to Drogba. United would give it to Ronaldo or Rooney. Liverpool would give it to Gerrard for a punt or lob it to Crouch for a shot (perhaps not that fast though).
Would Arsenal punt it up to Adebayor, or would Lehmann pass to Gallas, who'll give it to Silva, who'll pass to Hleb, who'll take 5 touches and slide-rule it to Fab, who'll pass to Rosicky for a shot?


The disclosure in September concerning a New Hampshirehospitalpatientwhodiedof Creutzfeldt- Jakob Disease (CJD) this past spring reaffirms the need for continuing and enhanced research in disease monitoring and disinfection techniques within hospital environments. CJD is a rapidly progressive neurodegenerative disease that is always fatal within one year of onset—there is no cure. The disease can only be diagnosed positively through autopsies. While relatively rare, the number of annual deaths associated with CJD in the U.S. has risen steadily from 250 in 2000 to more than 350 in 2009.
Standard hospital disinfection techniques cannot eradicate the prion that causes CJD from surgical instruments. As a result, 15 patients were possibly exposed to CJD—eight in New Hampshire, five in Massachusetts, and two in Connecticut. The Massachusetts and Connecticut patients were all treated using the same Medtronic guided imaging navigation system and associated surgical tools used in the New Hampshire surgery. Hospitals often share these high-cost neurosurgery tools on a rental basis, explaining the multi-state CJD exposure.
The CDC and the World Health Organization (WHO) have established guidelines for reprocessing surgical instruments potentially contaminated by CJD. These guidelines were derived from in vitro inactivation studies using brain tissues or tissue homogenates, both of which pose substantial challenges to any sterilization process. Obviously, destruction of the surgical devices is the safest and most unambiguous method, but often the most impractical and always the least cost-effective. According to the CDC, one of the three most stringent chemical and autoclave sterilization methods outlined in Annex III of the WHO guidelines should be used to reprocess heat-resistant instruments that come in contact with the highly infectious tissues of patients with suspected or confirmed CJD. Since these methods are extremely harsh and potentially corrosive, users should consult with the device manufacturers as to their instruments' tolerance of extended exposure to heated, concentrated sodium hypochlorite or sodium hydroxide—the common cleaning agents recommended in the WHO guidelines.
Following the New Hampshire CJD announcement, the Joint Commission, a non-profit hospital accreditation organization ( www.jointcommission.org), reiterated guidelines urging hospitals to have a high degree of suspicion about CJD even when diagnosis is not confirmed and to follow CDC/WHO guidance on sterilization or disposal of surgical tools. Mark Chassin, President/CEO of the Joint Commission, stated that too many hospitals experience serious safety failures as routine and inevitable parts of their daily work. He emphasized a framework for major changes involving leadership, safety culture and robust process improvement. The process improvement changes recommended include a combination of Six Sigma, Lean, and change management in order to make far greater progress toward eliminating patient harm.
Clearly, most of the safety methodologies recommended by the CDC, WHO, or the Joint Commission were not in place during the initial New Hampshire CJD surgery, otherwise the 15 patients would not have had to be informed of their deadly disease risk nearly six months after their possible exposure.
Consultant and Faculty Member, Clemson University Howard Siegerman, Ph.D.
Scott Sutton, PhD, Microbiology Network Inc. Art Vellutato, Jr. , Veltek Associates Inc.


The PiTop is a rather cool modular laptop that you build yourself and which is based around the Raspberry Pi platform. Eminently hackable, we thought it would be fun to see if we could integrate a LimeSDR Mini for software-defined radio development on the move. In this short video we show how we achieved this with the aid of just a few custom laser cut parts and give a quick demo using the excellent SDRangel software to view 2.4 GHz spectrum.


text:
Chin.J.Chem.Eng. ›› 2018, Vol. 26 ›› Issue (5): 922-929.doi: 10.1016/j.cjche.2017.10.026
• Fluid Dynamics and Transport Phenomena • Previous Articles Next Articles
Abstract: The interaction between fluid and a down-pumping pitched blade turbine fixed with a flexible shaft in the stirred vessel, as a typical fluid structure interaction phenomenon, was simulated by coupling the Computational Fluid Dynamics and Computational Structural Dynamics. Based on the verification of the simulated impeller torque and dimensionless shaft bending moment with experimental result, the dimensionless shaft bending moment and various loads acting on impeller (including lateral force, axial force and bending moment) were discussed in detail. By separating and extracting the fluid and structural components from those loads, the results show that the shaft bending moment mainly results from the lateral force on impeller although the axial force on impeller is much larger. The impeller mass imbalance increases the shaft bending moment and the lateral force on impeller, but has little influence on the axial force and bending moment acting on impeller. The dominant frequencies of impeller forces are macro-frequency, speed frequency and blade passing frequency, and are associated with the impeller mass imbalance.
Key words: Fluid structure interaction, Shaft bending moment, Impeller lateral force, Impeller axial force, Bending moment on impeller
Yangyang Liang, Zhengming Gao, Dai'en Shi, Wanli Zhao, Ziqi Cai. Coupling simulation of fluid structure interaction in the stirred vessel with a pitched blade turbine[J]. Chin.J.Chem.Eng., 2018, 26(5): 922-929.
http://cjche.cip.com.cn/EN/Y2018/V26/I5/922
[1] S. Roy, S. Acharya, M.D. Cloeter, Flow structure and the effect of macro-instabilities in a pitched blade stirred tank, Chem. Eng. Sci. 65(2010) 3009-3024.
[2] P. Hasal, I. Fort, J. Kratena, Force effects of the macro-instability of flow pattern on radial baffles in a stirred vessel with pitched-blade and Rushton turbine impellers, Chem. Eng. Res. Des. 82(2004) 1268-1281.
[3] K.V. Riet, W. Bruijn, J.M. Smith, Real and pseudo-turbulence in the discharge stream from a Rushton turbine, Chem. Eng. Sci. 31(1976) 407-112.
[4] R. Escudie, A. Line, Experimental analysis of hydrodynamics in a radially agitated tank, AICHE J. 49(2003) 585-603.
[5] X.H. Liu, Y.H. Bao, Z.P. Li, Z.M. Gao, J.M. Smith, Particle image velocimetry study of turbulence characteristics in a vessel agitated by a dual Rushton impeller, Chin. J. Chem. Eng. 16(2008) 700-708.
[6] D.E. Shi, Y.Y. Liang, A. Eaglesham, Z.M. Gao, Effect of the impeller imbalance on the bending moment acting on an overhung shaft in a stirred vessel, Chem. Eng. Res. Des. 92(2014) 2191-2200.
[7] N.R. Kippers, A.G.L. Holloway, Experiments on the whirling of pitched blade impellers in baffled mixing vessels, J. Fluid Struct. 49(2014) 29-52.
[8] D.E. Shi, T. Lu, A. Eaglesham, Z.M. Gao, Characteristics of the bending moment acting on an overhung shaft in a stirred vessel, Int. J. Chem. React. Eng. 12(2014) 135-150.
[9] G.L. Lane, G.D. Rigby, G.M. Evans, Pressure distribution on the surface of Rushton turbine blades-experimental measurement and prediction by CFD, J. Chem. Eng. Jpn 34(2001) 613-620.
[10] T. Berger, M. Fischer, K. Strohmeier, Fluid-structure interaction of stirrers in mixing vessels, Trans. ASME 125(2003) 440-445.
[11] T. Berger, K. Strohmeier, Numerical simulation of stirrer oscillations in consideration of fluid-structure-interaction and flexible restraint systems, Proceedings of the ASME Pressure Vessels and Piping Division Conference, AICHE J, Cleveland Ohio, 2003 pp.
[12] K.M. Mohamed, A.G. Gerber, G.A.L. Holloway, Modelling of hydrodynamic forces on a whirling mixing vessel stirrer including fluid-structure interaction, Proceedings of the ASME Pressure Vessels and Piping Division Conference, J. Chem. Eng. Jpn, Prague, Czech Republic, 2009 pp.
[13] G.C. Cudmore, A.G.L. Holloway, A.G. Gerber, Whirl instability of a rotating impeller in a baffled mixing vessel, Proceedings of the ASME Pressure Vessels and Piping Division Conference, J. Chem. Eng. Jpn, Paris, France, 2013(pp.).
[14] D.E. Shi, Z.Q. Cai, A. Eaglesham, Z.M. Gao, Coupling simulation of lateral fluid structure interaction in a stirred vessel with a Rushton turbine, J. Chem. Eng. Jpn 48(2015) 147-157.
[15] S. Karray, Z. Driss, A. Kaffel, H. Kchaou, M.S. Abid, Fluid-structure interaction in a stirred vessel equipped with a Rushton turbine, Int. J. Mech. Appl. 2(2012) 129-139.
[16] S. Karray, Z. Driss, A. Kaffel, H. Kchaou, M.S. Abid, Numerical simulation of fluidstructure interaction in a stirred vessel equipped with an anchor impeller, J. Mech. Sci. Technol. 25(2011) 1749-1760.
[17] Y.L. Young, Fluid-structure interaction analysis of flexible composite marine propellers, J. Fluids Struct. 24(2008) 799-818.
[18] C.Y. Khor, M.Z. Abdullah, F.C. Ani, Study on the fluid/structure interaction at different inlet pressures in molded packaging, Microelectron. Eng. 88(2011) 3182-3194.
[19] B. Landvogt, L. Osiecki, P. Patrosz, T. Zawistowski, B. Zylinski, Numerical simulation of fluid-structure interaction in the design process for a new axial hydraulic pump, Prog. Comput. Fluid Dyn. 14(2014) 31-37.
[20] C.H. Lim, M.Z. Abdullah, I.A. Azid, M.S.A. Aziz, Experimental and numerical investigation of flow and thermal effects on flexible printed circuit board, Microelectron. Reliab. 72(2017) 5-17.
Related Articles 2
[1] Yangyang Liang, Zhengming Gao, Dai'en Shi, Haotian Li, Yuyun Bao, Ziqi Cai. Torque and bending moment acting on a flexible shaft agitated by disk turbines in a gas-liquid stirred vessel [J]. Chinese Journal of Chemical Engineering, 2019, 27(4): 781-793.
[2] Dai'en Shi, Ziqi Cai, Archie Eaglesham, Zhengming Gao. Effects of bubbly flow on bending moment acting on the shaft of a gas sparged vessel stirred by a Rushton turbine [J]. , 2015, 23(3): 482-489.
[1] LI Deliang(李德亮), LIU Xiaoqiang(刘小强) and CUI Jiehu(崔节虎). Reactive Extraction of o-Aminophenol Using Trialkylphosphine Oxide[J]. , 2006, 14(1): 46 -50 .
[2] JINHaibo(靳海波),M.WangbandR.A.Williamsb. The Effect of Sparger Geometry on Gas Bubble Flow Behaviors Using Electrical Resistance Tomography[J]. , 2006, 14(1): 127 -131 .
[3] PatrickPERRE. Evidence of Dual Scale Porous Mechanisms During Fluid Migration in Hardwood Species (Ⅱ) A Dual Scale Computational Model to Describe the Experimental Results[J]. , 2004, 12(6): 783 -791 .
[4] JIANG Chengfa, ZHANG Yunxiang. Kinetic Study of Atom Transfer Radical Polymerization of 2-(N,N-Dimethylamino) ethyl
Methacrylate[J]. , 2004, 12(2): 208 -213 .
[5] LIU Zhichang, MENG Xianghai, XU Chunming, GAO Jinsen. Secondary Cracking of Gasoline and Diesel from Heavy Oil Catalytic Pyrolysis[J]. , 2007, 15(3): 309 -314 .
[6] HUANG Fuming, ZHANG Guowei, HU Chunpu, YING Shengkang. Preparation and Characterization of Hybrid Aqueous Dispersions Composed of Silica Sol and Poly(styrene-co-acrylate)[J]. , 2005, 13(6): 816 -823 .
[7] SUN Haixiang, ZHAN Lin, CHAI Hong, CHEN Huanlin. Removing Endotoxin from Protein Solution by Chitosan Modified Affinity Membrane[J]. , 2005, 13(4): 457 -463 .
[8] ZHANG Yu, ZHOU Lixing, WEI Xiaolin, SHENG Hongzhi. Numerical Simulation of NOx Formation in Coal Combustion with Inlet Natural Gas Burning[J]. , 2005, 13(3): 318 -323 .
[9] JI Zhongli, JIAO Haiqing, CHEN Honghai. Image Analysis on Detachment Process of Dust Cake on Ceramic Candle Filter[J]. , 2005, 13(2): 178 -183 .
[10] WU Yanxiang, WANG Biyu. Analysis of the Medium Resistance for Constant Pressure Filtration[J]. , 2004, 12(1): 33 -36 .


This is a fascinating look into the world of a saturation diver.
Hovey works in one of the least known, most dangerous, and, frankly, most bizarre professions on Earth. He is a saturation diver—one of the men (right now they are all men) who do construction and demolition work at depths up to 1,000 feet or more below the surface of the ocean.


Hillsville Hardee's Biscuit Maker Wins Top Award in Biscuit Maker Challenge - Boddie-Noell Enterprises Inc.
HILLSVILLE,VA. (Feb. 26, 2014) – If the breakfast biscuits baked at the Hardee's® restaurant at 115 N. Main St. taste delicious, that's because they are made by a Biscuit Maker Champion.
Local Hardee's biscuit maker Mary Marshall was named this week as the 2013 Best Biscuit Maker of Boddie-Noell Enterprises, the largest Hardee's franchise operator in the nation. Marshall advanced beyond more than 700 other biscuit makers at 335 Hardee's restaurants across four states to win the overall award.
Contestants in Boddie-Noell's annual Biscuit Maker Challenge represented Hardee's restaurants across North Carolina, Virginia, South Carolina and Kentucky. In addition to Marshall, the three best biscuit maker regional winners were Elijah Hoff of the Hardee's in Timberville, Anthony Robinson of Virginia Beach and Jeanette Thomas of Aberdeen, N.C.
Boddie-Noell honored the four regional winners and announced Marshall as the overall winner during a banquet Sunday night at the company's Rose Hill Plantation near Nashville, NC.
Marshall and the three other regional winners will be featured later this year in photos on the in-store tray liners at all Hardee's restaurants operated by Boddie-Noell.
The competition started in October, and each biscuit maker was evaluated based on the quality of his or her biscuits, knowledge of proper biscuit equipment, cleanliness and care of the biscuit equipment, knowledge and execution of company procedures, grooming and appearance. The competition's judges included general managers, district managers, regional directors, vice presidents and other executives in the restaurant division of Boddie-Noell at five levels of competition.
Marshall, who lives in Hillsville, has worked as a biscuit maker at Hardee's for more than 5 years. Marshall says the key to creating the best Hardee's Made from Scratch™ Biscuit begins with "caring about what you do." After that, being consistent and "following procedures every day and practice, practice, practice" is key. Marshall also noted she has been baking her whole life. Her favorite biscuit? "The Sausage Biscuit with grape jelly," she said.
Aside from making biscuits, Marshall enjoys baking, particularly pound cakes and German chocolate cake for her son and his wife as well as her four grandchildren. She doesn't mind rising early at 4 a.m. to begin baking biscuits for customers as she's always been an early riser. Her family talked her into starting biscuit baking based on her longtime love of making foods that family and friends alike enjoyed.
Hardee's Made from Scratch™ biscuits, which have been served at Boddie-Noell restaurants since 1977, are a vital part of the restaurant's menu.
In 2013, Hardee's restaurants owned and operated by Boddie-Noell used more than 10.8 million pounds of flour and biscuit mix and 867,000 gallons of buttermilk to make the golden brown biscuits – enough flour, mix and buttermilk to fill more than 47 tractor-trailers.
With over 50 years success as a Hardee's franchisee, family-owned Boddie-Noell is the largest Hardee's franchise operator in the United States with 335 restaurant locations in four states. The company is based in Rocky Mount, N.C. For more information, visit www.bneinc.com.


We project former Yankee starter Mike Mussina to narrowly miss a ticket to Cooperstown.
We analyze elections here at FiveThirtyEight, and we're not about to miss one — even if it takes place in the world of baseball. On Tuesday, the Hall of Fame's full Class of 2019 will be announced, and Mariano Rivera, Edgar Martinez and more than 30 other former players will find out if they will join a list of immortals in baseball's most exclusive club.
Hopefuls earn a spot in the Hall if they're named on at least 75 percent of ballots cast by typically around 400 voting members of the Baseball Writers' Association of America.1 (If not, they roll over to next year's ballot — unless they get fewer than 5 percent of the vote or have been on the ballot for 10 years.) We already have some idea of what percentage to expect for each player, thanks to the efforts of Ryan Thibodaux. Thibodaux is the exit pollster of the baseball world: He and his team of "interns" scour the internet for voters who have made their Hall of Fame ballots public before the full results are announced. Thibodaux tallies them all up on his online Baseball Hall of Fame Vote Tracker — like the first few precincts to report on election night — to give us partial election results weeks in advance.
The problem is, as experienced election-watchers know, the early returns may not be representative of the whole electorate. It turns out that the public vote is consistently higher than the actual vote for some players and lower for others. In pollster lingo, this is because of sampling bias. Voters who are more likely to publish their ballots in advance are those more likely to be on Twitter (the main medium for sharing ballots) and vote a certain way. As the table below demonstrates, players who are favored by the sabermetric crowd (for example, Tim Raines and Mike Mussina) or tied to performance-enhancing drugs (Barry Bonds, Roger Clemens) historically are overrepresented on public ballots, while players perceived as "clean" (Fred McGriff) or whose cases rely on traditional statistics like saves or Gold Glove Awards (Omar Vizquel, Lee Smith) do better on private ballots.
Positive numbers mean the player did better on private ballots than public ballots; negative numbers mean the player did better on public ballots than private ballots.
*Eventually elected to the Hall of Fame by the Baseball Writers' Association of America.
Excludes candidates who have received zero votes on public ballots.
*The adjustment factor is derived from historical differences between public and private ballots.
I'm projecting that three players will surpass the 75 percent threshold required for election. Let's start with the player who has waited the longest to get here: Seattle Mariners hitting machine Edgar Martinez, who is eligible for election for the 10th and final time.3 Martinez is currently pulling 90.7 percent of the public vote, but — perhaps because of anti-designated-hitter bias — he is one of the candidates who always see precipitous drops from the exit polls to final results. Still, I'm expecting him to win 73.9 percent of the votes that have yet to be revealed, giving him an overall total of 81.3 percent.
With a projected 92.6 percent of the vote, fearsome starting pitcher Roy Halladay also looks assured of election. This is the first time Halladay — who tragically died in a plane crash in 2017 — has appeared on the ballot. That means he has no vote history to go off, so my model treats him (and other first-time candidates) a little differently. First, I calculate which returning candidate's support is most highly correlated with the new candidate's support. For Halladay, this was Mike Mussina, which makes sense — both are starting pitchers who fall short of the traditional Hall of Fame standard (300 wins) but have strong peripheral stats. Then I estimate the new candidate's support among private ballots assuming that correlation will hold steady. For example, so far in public ballots, Halladay has won 96.6 percent of voters who also voted for Mussina but only 81.8 percent of voters who didn't vote for Mussina. I applied those same percentages to the projected number of private ballots with and without Mussina.
Mariano Rivera — indisputably the best closer of all time — rounds out this group.4 Currently, I'm projecting him to get a round 100 percent of the vote, which would make him the first unanimous selection in Hall of Fame history. However, I freely admit that candidacies like Rivera's — another first-time candidate — are a blind spot for the model. With no public "no" votes on Rivera, he correlates equally poorly with every returning candidate.5 Therefore, our best option is to look at how other closers, like Trevor Hoffman and Lee Smith, have fared — and if there is one ironclad rule for the public-private differentials, it's that closers do even better with private voters than they do with public ones. You can't do better than 100 percent, obviously, so Rivera's adjustment factor is simply zero.
The most interesting finding belongs to a candidate who isn't currently projected to make the Hall: Mussina. Although his name has been checked on 81.9 percent of ballots revealed so far, he historically has performed about 16 points worse among private voters. At the current ratio of public-to-private ballots, that computes to a final vote share of 73.0 percent — just 2 points shy of election. But if there's one thing you should have learned from reading FiveThirtyEight, it's that polling errors of 2 points happen all the time, so Mussina could definitely still pull this out. Even if Mussina doesn't make it this year, though, he's a cinch to get in eventually, as he has four years of eligibility remaining.
No other candidates are particularly close to being elected, according to my projections, although that doesn't mean their forecasted vote shares aren't interesting. If you just take Thibodaux's tracker as gospel, you might think Barry Bonds (73.1 percent of public ballots) and Roger Clemens (73.6 percent) are knocking on the door. But the two superstars, both rumored to have been heavy steroid users, are virtually guaranteed to bomb with private voters. Their projected percentages in the low 60s are only incremental improvements from their 2018 vote shares, creating genuine suspense over whether they will be elected before their window of opportunity expires in 2022. On the other hand, Larry Walker is forecasted to finish at 62.2 percent, which would be a 28.1-point improvement over last year — a record. Amazingly, that would mean the ex-Rockies outfielder, whose candidacy seemed stuck in purgatory just last year, would have a real shot at election in 2020, his final year on the ballot.
Finally, the exit polls imply that a few candidates — including Andruw Jones and Andy Pettitte — are in danger of falling off the ballot entirely. But my model thinks they'll live to be voted upon another day. Last year, Jones won private ballots at twice the rate that he won public ballots, and first-time candidate Pettitte's support is negatively correlated with Martinez's support, implying that he'll gain on private ballots as well. The only candidates I'm expecting to drop off are those whose support in Thibodaux's tracker is so limited that it would take a miracle among private ballots to save them. Sorry, Michael Young, Lance Berkman, Roy Oswalt and Miguel Tejada.
Specifically, those who have covered baseball for at least 10 of the last 20 years. Each Hall voter can pick up to 10 candidates listed on the ballot.
Specifically, the adjustment factor is an average of each player's last two public-private differentials, with his 2018 differential weighted 75 percent and his 2017 differential weighted 25 percent. If you've followed my projections in the past, you might notice that this is different from how I used to calculate adjustment factors, which was a straight average of the player's last three differentials. After conducting an autopsy of last year's projections, I found that this weighted two-year average has historically been more accurate.
In 2014, the Hall of Fame announced that players would lapse off the ballot after 10 years, not 15 as in the past.
Harold Baines and Lee Smith were chosen for Hall induction late last year as part of an alternative election process.
The same is true for candidates with no public "yes" votes, like Kevin Youkilis. I don't issue projections for these "just happy to be nominated" candidates.


This beautiful home is located in Falconhead. Features tile & hardwoods floors, granite counters, top of the line appliances, butlers pantry and oversized laundry room w sink. Open floor plan includes a spacious master bedroom, double staircase, formal dining, beautiful sunroom/office with it's own patio and full bath. Spacious covered patio and large fenced backyard. Situated on the 5th hole of Falconhead Golf Course.


text:
\section{Introduction}

Regularized estimators such as Lasso~\cite{Tibshirani-LASSO}, Square-Root Lasso~\cite{Belloni11,Stadler10,ScaledLasso11}, MCP~\cite{Zhang10}, and SCAD~\cite{Fan_Li01} hinge on tuning parameters. However, standard calibration schemes for these tuning parameters are computationally infeasible or intractable for finite sample theory. As a consequence, the calibration of regularized estimators is a crucial, unsolved problem.

We focus on the calibration of Lasso for linear regression, where the tuning parameter needs to suit both the noise distribution and the design matrix~\cite{vdGeer11,YoyoMomo12,ArnakMoYo14}. Calibration schemes for this setting are typically based on Cross-Validation or BIC-type criteria. However, Cross-Validation requires extensive computations and lacks for finite sample guarantees. BIC-type criteria, on the other hand, are computationally simpler but also lack for finite sample guarantees. Another approach is to replace Lasso with Square-Root Lasso or TREX~\cite{Lederer:14}; however, Square-Root Lasso still contains a tuning parameter that needs to be calibrated to certain aspects of the model, and the theory for TREX is currently fragmentary. Therefore, new insights into the calibration of Lasso are necessary.

In this paper, we introduce novel calibration schemes for minimal sup-norm loss of Lasso: Adaptive Validation for $\ell_\infty$ (\avi) and Adaptive Validation With Multiple Constants (\avim). \avi~and \avim~consist of tests that are inspired by Lepski's method for nonparametric regression~\cite{Lepski90,Lepski_Mammen_Spokoiny97} and provide, in strong contrast to standard calibration schemes, both {\bf optimal theoretical guarantees} and {\bf fast computations}. In Section~\ref{methodology}, we introduce \avi~and \avim~and present the following:\vspace{-0.75mm}
\begin{itemize}
\item {Optimal} finite samples guarantees for the calibration of Lasso with respect to sup-norm loss in a broad class of linear regression models (Theorem~\ref{th} and Theorem~\ref{thmf});\vspace{-1.5mm}
\item A safe thresholding procedure to reduce false positives (Remark~\ref{thre});\vspace{-1.5mm}
\item A {simple and fast} algorithm (Algorithm~\ref{foxalgo}).
\end{itemize}
\vspace{-0.6mm}These results show that \avi~and \avim~can maximize the Lasso performance for sup-norm loss  and variable selection. In Section~\ref{simulations}, we then support these results with applications to synthetic data and to biological data, and  in Section~\ref{conclusions}, we finally conclude with a discussion of our approach.

\section{Methodology}\label{methodology}

\subsection{Framework}

redpajama_set_name: RedPajamaArXiv