text:
\section{Introduction}
\label{sec:introduction}
In order to survive, any living organism has to adapt to its environment.
Far from being static, the environment for most organisms is subject to constant change.
Arguably, complex and plastic information processing structures such as the mammal brain are an evolutionary answer to the requirement to adapt to unforeseen circumstances during the lifespan of the organism~\cite{allen2012lives}.

Some aspects of this adaptive behaviour, most notable in higher mammals such as humans, are called \emph{creative}.
Although the term is notoriously evasive of a precise definition, there is common agreement that creative behaviour involves elements such as novelty, surprise, and value \cite{weisberg1993creativity,csikszentmihalyi1996creativity}.
An example of creative behaviour can be found in the use of domain names in the world wide web.
Although Top Level Domains (such as \emph{.com}, \emph{.org}, \emph{it}) are intended as a means of structuring the world wide web, they are nowadays used for their linguistic meaning, rather than their original denotation of thematical or geographical structure.
Examples of such uses are \emph{youtu.be}, \emph{friend.ly}, and \emph{podca.st}.
Arguably, this creative use is driven by the increasing use and communication of URLs, and therefore a growing need for shorter URLs, that are easy to memorize.

One of the most seminal attempts at formalizing the notion of creativity is that of \citet{Boden:2004wl}.
She links creative behaviour of an agent to its \emph{conceptual space}.
This space represents the way the agent interprets its perception and actions.
According to Boden, creativity amounts to mapping, exploration, and transformation of a conceptual space by an agent.
An example of such a transformation given by \citet[p.
522]{boden1994precis} is the transformation of the conceptual space for music by Arnold Schoenberg, who invented radically new music by dropping the constraint of the home key from the existing conceptual space of music.

Most literature on creativity focuses on the high level, cognitive phenomena it involves, taking for granted that a conceptual space, a way of dealing with the environment, is already present.
In contrast to viewing creativity as a discrete transformation of one static conceptual space into another, a promising alternative perspective is to regard conceptual spaces as inherently dynamic.
In this view, creativity is an inherent property of the dynamics of conceptual spaces that brings them into existence in the first place.
This view also suggests a more gradual distinction between perception and cognition.
\citet[p.
308]{hofstadter2008fluid} goes so far as to say that the ability to \emph{reperceive} is a critical element of creativity.

This focus on the dynamic aspects of conceptual spaces naturally leads to \emph{representation learning}~\cite{bengio13:_repres_learn}, an active field of research in machine learning.
In this area, computational models, predominantly in the form of neural networks, are trained on data to form hierarchically structured representations of the data.
Often, such representations capture semantically relevant characteristics of the data at several levels, and as such form a robust basis for subsequent tasks such as classification, or organization of data.

Typical methods for representation learning deal with the classical machine learning scenario in which a model is trained on a set of data $\ensuremath{\mathbf{x}}$, possibly with labels $\ensuremath{\mathbf{y}}$, drawn from a distribution $p(\ensuremath{\mathbf{x}}, \ensuremath{\mathbf{y}})$ (the generating distribution), and evaluated on further data that is also assumed to be from the same distribution $p(\ensuremath{\mathbf{x}}, \ensuremath{\mathbf{y}})$.
This implies that the training methods are designed to converge to a single set (or hierarchical structure) of representations, that is optimal given $p(\ensuremath{\mathbf{x}}, \ensuremath{\mathbf{y}})$.
As stated at the beginning of this Section however, creativity is often driven by a need to adapt to a changing environment.
In terms of the machine learning paradigm, this violates the assumption that $p(\ensuremath{\mathbf{x}}, \ensuremath{\mathbf{y}})$ is static over time.

Thus, for a representation learning model to stay effective in the face of a changing environment requires methods beyond standard representation learning algorithms.
This shift of focus from learning representations in a static environment to adapting learned representations in a dynamic environment brings on multiple non-trivial problems to be addressed.
For example, there is a need for a measure of how well a given representation suits the environmental requirements.
Furthermore, in the light of an environmental change, sometimes gradual change of the learned representations may be beneficial, whereas on other occasions, a more radical change (for example by learning representations from scratch) may be more effective.

It is not in the scope of the current report to address all of these problems.
In this document, we restrict ourselves to the scenario where a radical change in environment is given, and investigate which of several alternative strategies for dealing with that change are most effective in neural network models for representation learning.
We are ultimately interested in finding generally useful strategies that allow a computational system to adapt efficiently to changing environmental requirements.
We start by a precise description of the problem, and the model architecture we use for learning.
Then we define a number of possible adaptation strategies, and report on an experiment in which these strategies are compared by evaluating them on several data sets.
Before that, we discuss how the work presented here relates to creativity in a broader sense.

\subsection{From adaptive representations to creative behaviour}\label{sec:from-adapt-repr}
In this document, we focus on strategies for conceptual change to adapt learned representations from one task to another task.
The tasks we consider here are classification, and autoencoding.
A change of task (for the model to adapt to) may either mean switching from classification to auto-encoding or vice versa, or in the case of classification, a change of target classes.
Note however, that the change of task inducing the conceptual change (the changes in the learned representations) in these cases is an extrinsic factor.
From the perspective of an agent interacting with an outside world, such changes may reflect changes in its environment (e.g.
a change of habitat, with different species and objects).
This scenario corresponds to the type of creativity described above, a form of adaptive, problem solving behaviour.
A different, but related interpretation emphasizes the generative, aesthetic aspects of creative behaviour, that may lead to novel artefacts.
In this case, the conceptual change manifest in the creative behaviour is thought to be driven by intrinsic factors rather than as a response to changes in the environment.

Are the methods for conceptual change described here, and evaluated in the context of classification and autoencoding, also of use in the realization of this second kind of creativity?
Two current computational theories of creative behaviour suggest they may be, since both posit that at the basis of creative behaviour are mechanisms that encode incoming information, and adapt to a dynamic environment to provide optimal encodings, in line with evidence for learning principles in living organisms, such as \emph{minimum entropy coding} \cite{barlow89:_unsup_learn,atick11:_could,olsh96}.
We will briefly describe both theories.

\citet{wiggins15:_idyot} propose a theory of creativity based on the Global Workspace Theory~\cite{baars2002conscious}.
The criterion that determines what representations develop in this model is based on information-theoretic principles, demanding that the representations facilitate accurate prediction of the perceptual future, leading to more efficient information processing.
In their model, multiple competing \emph{generators} match perceptual input to learned representations from memory to predict future input. The representations most of the effective generators surface into the \emph{Global Workspace}, reflecting \emph{conscious awareness}.
This awareness in turn, updates an associative memory that stores the adaptive representations to be used by the generators.
The changes in semantics as a result of adapting the conceptual space (for instance in the form of \emph{aberration} \cite{wiggins06:_prelim_framew_descr_analy_compar_creat_system}) may be regarded as a restricted form of creative behaviour. In addition, \citet{wiggins15:_idyot} theorize that an instantiation of the theory in the form of a computational model, after being exposed to stimuli that have imprinted memories, is able to generate novel artefacts: In the absence of external stimuli, the system can fill its perceptual buffers from memory, thus continuing the cycle of generation from memory, selection, and updating memory.

\citet{schmidhuber10:_formal_theor_creat_fun_intrin_motiv} also argues that a fundamental aspect of cognition in an agent is to learn to compress (or equivalently, predict), the outside world and its affordances.
Plausible drivers for this goal of optimal compression/prediction are extrinsic rewards such as sparing use of physiological resources (information that can be sparsely represented takes less energy to process and retain), and a selective advantage (by allowing better anticipation of future events).
But Schmidhuber argues that an important explanatory factor for human behaviour is an \emph{intrinsic} motivation to \emph{improve} compression/prediction capabilities.
This means that for an agent it is desirable in itself to discover ways of better compressing and predicting its experience, where the (intrinsic) reward is proportional to the degree of improvement.
Schmidhuber claims that this intrinsic motivation manifests as \emph{curiosity} in human beings, and that the subjective experience of improving compression/prediction is aesthetic (that is: fun, beauty, or elegance).
This also leads to a notion of surprise that is different from the usual information-theoretic notion of surprise, in the sense that rather than the information content of an event itself, it is the degree to which the information content of an event triggers an improvement of the agent's compression/predicting capabilities.
Thus, just as a sequence of repeated events (no information content), a sequence of random events (high information-content) is not surprising, and will soon become boring.

Both of the theories of creative behaviour described above involve an adaptive component that generates, predicts, or compresses perceptual input, in other words, it maps incoming data to an internal representation that adapts to accommodate novel patterns in the data.
The precise nature of this accommodation process is not part of the above (rather high level) theories, but the methods evaluated here hint at possible implementations of such a process, especially in situations where the patterns in the data change drastically.
Furthermore, the fact that the methods proposed here concern (deep) neural networks suits the \emph{reinforcement learning} (RL) paradigm of Schmidhubers theory, where neural networks in combination with Q-learning \cite{watkins1989learning} have been shown to be very successful at a variety of game-playing tasks~\cite{2013arXiv1312.5602M,2015arXiv150906461V}.

The remainder of this document is structured as follows.
In Section~\ref{sec:related-work}, we discuss a number of related problems known from machine learning and discuss their relevance for the problem at hand.
In Section~\ref{sec:method}, we introduce the architecture of the neural network model we use, and a number of possible adaptation strategies to facilitate conceptual change.
Section~\ref{sec:experiment} presents the setup of the comparative evaluation of the strategies.
The results of this evaluation are reported, and discussed in Section~\ref{sec:resanddisc}.
Conclusions are presented in Section~\ref{sec:conclusions}.

\section{Related work}\label{sec:related-work}

\subsection{Transfer learning}\label{sec:transfer-learning}
\input{related_work/transfer_learning}

\subsection{Domain adaptation}\label{sec:domain-adaptation}
\input{related_work/domain_adaptation}

\subsection{Multi-task learning}\label{sec:multi-task-learn}
\input{related_work/multi_task_learning}

\subsection{Concept drift}\label{sec:concept-drift}
\input{related_work/concept_drift}

\section{Method}\label{sec:method}

The specific problem we focus on in the current experiment is a form of transfer learning.
We are interested in the question how a model that has been trained to perform a particular task, can adapt to a novel task most effectively.
As illustrated in Figure~\ref{fig:transfer_plot}, there are several aspects to the notion of effective adaptation --- most importantly the asymptotic accuracy on the novel task, and the amount of training it takes to perform a task accurately.
We define the criteria we use for evaluation adaptation strategies in Section~\ref{sec:evaluation-criteria}, but before that, we will give a more precise description of the problem we are addressing.

\subsection{Problem description}\label{sec:problem-desc}

Given a labelled data set, we divide the data set into two subsets, such that one subset contains the data pertaining to one half of the labels, and the other subset contains the data pertaining to the other half of the labels.
We call one subset the \emph{source} domain, and the other the \emph{target} domain.
This implies that the label space is different, i.e.
$\ensuremath{\mathcal{Y}}_s \neq \ensuremath{\mathcal{Y}}_t$, and therefore $P_s(Y \mid X) \neq P_t(Y \mid X)$.
Although splitting the data set by labels does not strictly imply that the marginal data distributions of the domains are different, the labels usually relate to some morphological aspect of the features, so it is likely that  $P_s(X) \neq P_t(X)$.

The tasks we consider are classification, and autoencoding, and multi-task learning, in which a model is simultaneously trained to perform classification and autoencoding.
The classification task is formalized as $\ensuremath{\mathcal{T}}^{CL} = \{ \ensuremath{\mathcal{Y}}, P(Y \mid X) \}$.
Note that in the autoencoding task, the labels $Y$ are not used.
Unfortunately, the probabilistic formalism is not well-suited to formalize the autoencoding task: autoencoding would be regarded as a case where $\ensuremath{\mathcal{Y}} = \ensuremath{\mathcal{X}}$, which implies the trivial task $\ensuremath{\mathcal{T}}^{AE} = \{ \ensuremath{\mathcal{X}}, P(X \mid X) \}$.
The actual value of the autoencoding task is in the fact that the \emph{bottleneck} architecture of the models used to approximate $P(X \mid X)$ prevents them from learning a trivial mapping.
A more precise description of the autoencoding task is given in Section~\ref{sec:convae}.
Finally, multi-task learning involving simultaneous classification and autoencoding can be formalized as follows.
Instead of the label space \ensuremath{\mathcal{Y}}, we define $\ensuremath{\mathcal{Z}} = \ensuremath{\mathcal{X}} \times \ensuremath{\mathcal{Y}}$.
The multi-task formalization is then $\ensuremath{\mathcal{T}}^{MT} = \{ \ensuremath{\mathcal{Z}},  P(Z | X) \} $, where $Z \in \ensuremath{\mathcal{Z}}$.

The aim of an adaptation strategy is then to transform a model that is optimal for task $\ensuremath{\mathcal{T}}_s$ on $\ensuremath{\mathcal{D}}_s$ into a model that is optimal on $\ensuremath{\mathcal{T}}_t$ on $\ensuremath{\mathcal{D}}_t$.
For example, a model trained for a classification task $\ensuremath{\mathcal{T}}^{CL}$ in domain $\ensuremath{\mathcal{D}}_s$, might be adapted to perform a classification task $\ensuremath{\mathcal{T}}^{CL}$ in domain $\ensuremath{\mathcal{D}}_t$, but it may also be adapted more radically, to perform autoencoding $\ensuremath{\mathcal{T}}^{AE}$ in domain $\ensuremath{\mathcal{D}}_t$.

Since the architecture of a model is determined by the task it performs, it is not obvious what we mean by adapting a model from one task to another.
In Section~\ref{sec:transfer-cnn-model} we will describe this procedure in more detail.
Another aspect to be clarified is how we measure the success of adaptation strategies.
We address this issue in Section~\ref{sec:evaluation-criteria}.

\subsection{Evaluation criteria}\label{sec:evaluation-criteria}
As described above, our goal is to find adaptation strategies for representation learning models that allow a model to perform as well as possible on a new task, with as little training as necessary.
In terms of the schematic plot in Figure~\ref{fig:transfer_plot}, our primary interest is in the slope of the learning curve, but obviously, a model that adapts quickly but has substantially lower asymptotic performance than is possible with a baseline method is undesirable.
Therefore, both the slope and the asymptotic performance with respect to some baseline methods should be taken into account when evaluating the methods.

We are ultimately interested in adaptation strategies that work well independent of the source and target tasks involved.
This suggests that we define a single quality measure, that aggregates the the slopes and asymptotic performances of a strategy over the different combinations of source and target tasks.
We chosen not to do so for the following reasons.
Firstly, since they are different quantities, there is no obvious way to combine the slope and asymptotic performance of an adaptation strategy in a principled manner.
A weighting of the two quantities necessarily reflects a personal judgement on their relative importance.
Secondly, the experiments reported here should be regarded as explorative.
Although we are interested in establishing the superiority of one adaptation strategy over another, independent of the tasks and domains involved, we have no a priori indication that such a ranking can be established across tasks and domains.

For these reasons, we refrain from defining a single evaluation criterion for the adaptation strategies.
We believe that more insight is gained by a qualitative analysis of the evolution of task performance in target domain for each of the adaptation strategies, as a function of training, using plots similar to the diagram in Figure~\ref{fig:transfer_plot}.

\subsection{Convolutional Neural Networks} \label{sec:cnn}
Convolutional Neural Networks (CNNs) are a special kind of Feed Forward Neural Networks (FFNNs) that build some invariance properties into the structure of the neural network~\cite{Bishop:2006}.
CNNs have been successfully used in several machine learning applications, including natural language processing and image classification~\cite{Krizhevsky:2012wl}.
CNNs have a number of advantages over fully connect FFNNs. Firstly, the convolutional nature of the architecture, using small convolutional filters enforces the extraction of local features~\cite{Lecun:1998hy}.
Secondly, they typically have shared weights, which greatly reduces number of parameters compared with similar sized FFNNs~\cite{Krizhevsky:2012wl}.
Lastly, they typically perform spatial sub-sampling, which adds robustness against noise and local distortions.
A typical CNN has three building blocks: convolutional, subsampling, and dense (fully connected) layers.
A CNN is illustrated in Figure \ref{fig:cnn_architecture}.

The basic building block of CNNs are the convolutional layers.
The input of $l$-th convolutional layer consists of $m_1^{(l-1)}$ feature maps from the previous layer, while its output consists of $m_1^{(l)}$ feature maps.
The $i$-th feature map in this layer is given by
\begin{equation}
\mathbf{y}_i^{(l)} = f_l\left(\sum_{j=1}^{m_1^{(l-1)}} \mathbf{W}^{(l)}_{i,j}*\mathbf{y}_{j}^{(l-1)}  + \mathbf{B}_{i}^{(l)}\right),
\end{equation}
where $*$ represents the convolution of $\mathbf{y}_{j}^{(l-1)}$ with $\mathbf{W}_{i,j}^{(l)}$, a kernel of size $h_1^{(l)}  \times h_2^{(l)}$ connecting the $j$-th feature map in layer $(l-1)$ with the $i$-th feature map in layer $l$, $\mathbf{B}_i^{(l)}$ is a bias (matrix) and $f_l$ is an elementwise non-linear activation function.

Pooling can be understood as a form of non-linear downsampling.
Max-Pooling layers partition an input feature map into a set of non-overlapping rectangles (\emph{pool}), and for each such pool, outputs its maximum value.

Finally, dense layers are the standard fully connected layers in FFNNs.
The output of the $l$-th dense layer can be computed as
\begin{equation}
\mathbf{y}^{(l)} = f_l\left(\mathbf{W}^{(l)} \mathbf{y}^{(l-1)} + \mathbf{b}^{(l)} \right),\label{eq:fully_connected}
\end{equation}
where $\mathbf{W}^{(l)}$ is a filter connecting layer $(l-1)$ to layer $l$, $\mathbf{b}^{(l)}$ is a bias vector and $f_l$ is an elementwise non-linear activation function.
The set of all parameters of a CNN, i.e.\ kernels, filters and biases, will be denoted \ensuremath{\boldsymbol\theta}.

Common activation functions for CNNs include sigmoid, $\tanh$,  rectifier, and softmax, which is particularly useful as the output of a multi-class classifier~\cite{Bishop:2006}. See Appendix \ref{sec:act_functions} for explicit definitions of these activation functions.

In practice, convolutional and pooling layers are used to learn a feature hierarchy, while dense layers are used for classification purposes based on the computed features \cite{Lecun:1998hy}.
In the following, we will refer to the stack of a convolutional and pooling layers as a \emph{convolutional stage}, and the stack of dense layers as fully connected stage, or as a classification stage, if its primary objective is classification.

The classification task for CNNs can be formally described as follows.
Given a set of input images $\{\mathbf{x}_1, \dots, \mathbf{x}_N\}$, and a set of targets $\{ \mathbf{t}_1, \dots, \mathbf{t}_N\}$, where $\mathbf{t}_i$ is a one-hot encoding of the class of $\mathbf{x}_i$, the parameters of a CNN can be learned in a supervised way as
\begin{equation}
\hat{\ensuremath{\boldsymbol\theta}} = \argmin_{\ensuremath{\boldsymbol\theta}} L (\ensuremath{\boldsymbol\theta}),\label{eq:parameter_learning}
\end{equation}
\noindent where $L(\ensuremath{\boldsymbol\theta})$ is the loss function.
The standard loss function for a multi-class classification problem is the  mean categorical cross entropy\footnote{See Eq.~\eqref{eq:CCE} in Appendix \ref{sec:cost_functions}.}~\cite{Bishop:2006}.

\begin{figure}
\begin{center}
\begin{tikzpicture}
\input{figs/CNN.txt}
\end{tikzpicture}
\caption{Example of the architecture of a Convolutional Neural Network classifier, including two convolutional-pooling substages (layers $l_1$ and $l_2$, and $l_3$ and $l_4$), and a classification stage consisting of two fully connected layer (as a coding substage $l_5$, and the proper classifier $l_6$).} \label{fig:cnn_architecture}
\end{center}
\end{figure}

\subsubsection{Convolutional autoencoders} \label{sec:convae}
Most methods in unsupervised learning are based on the ``encoder-decoder'' paradigm~\cite{Masci:2011tz}, where the input is first transformed into a (typically) lower-dimensional space (\emph{encoding stage}) and then expanded to reproduce the initial data (\emph{decoding stage}).
Examples of this paradigm include Low-Complexity Coding and Decoding Machines, Predictability Minimization layers, Restricted Boltzmann Machines and autoencoders~\cite{Masci:2011tz}.

An autoencoder (AE) is a particular neural network architecture used for feature learning.
Its aim is to learn an encodings, i.e.,  distributed (and usually compact) representation of a set of data.
 Formally, an AE takes an input $\ensuremath{\mathbf{x}} \in \mathbb{R}^{N_x}$ and maps it to a latent  representation or \emph{encoding} $\mathbf{y}_h \in \mathbb{R}^{N_h}$, using a deterministic function of type
\begin{equation}
\mathbf{y}_h = f\left(\ensuremath{\mathbf{x}} ; \ensuremath{\boldsymbol\theta} \right) = f\left( \mathbf{W} \ensuremath{\mathbf{x}} + \mathbf{b} \right)
\end{equation}
where $\ensuremath{\boldsymbol\theta}=\{ \mathbf{W}, \mathbf{b}\}$ are the encoding parameters.
A typical autoencoder uses functions similar to those of the fully connected layers (see Eq.~\eqref{eq:fully_connected}).
This code is used to reconstruct the input by a reverse mapping (\emph{decoding}), i.e.,
\begin{equation}
\mathbf{y}_{decoding} = f\left(\mathbf{y}_h; \ensuremath{\boldsymbol\theta}^{'} \right) = f\left(\mathbf{W}^{'} \mathbf{y}_h + \mathbf{b}' \right),\label{eq:decoding_fully_connected}
\end{equation}
where $\ensuremath{\boldsymbol\theta}^{'}=\{\mathbf{W}^{'}, \mathbf{b}^{'}\}$ are the decoding parameters.
A usual constraint of the decoding parameters if for the weights to take the form $\mathbf{W}^{'} = \mathbf{W}^T$, i.e., use the same weights for encoding of the input and decoding of the latent representation.

Convolutional Autoencoders (CAEs) are (deep) autoencoders that use CNNs in the in the encoding stage~\cite{Masci:2011tz}.
As previously stated in Section \ref{sec:cnn}, CNNs allow for discovery of localized features that appear over 2D input.
In a CAE, the reconstruction of the input data is due to a combination of basic image patches based on the latent code.

The decoding layers corresponding to fully connected layers are given in the same form as in Eq.~\eqref{eq:decoding_fully_connected}.
For convolutional layers, the corresponding $i$-th feature map of decoding layer $l$ is given by
\begin{equation}
\mathbf{y}^{(l)}_i = f\left(\ensuremath{\mathbf{y}}^{h_{(l-1)}}; \ensuremath{\boldsymbol\theta}^{'} \right) = f\left(\sum_{j=1}^{m_l^{h_{(l-1)}}} \tilde{\mathbf{W}}_{i,j}^{(l)}  * \mathbf{y}_j^{h_{(l-1)}} + \mathbf{B}'^{(l)}\right),
\label{eq:decoding_convolutional}
\end{equation}
where $\ensuremath{\mathbf{y}}^{h_{(l-1)}}$ represents all $m_l^{h_{(l-1)}}$ feature maps of the encoding layer  $h_{(l-1)}$, $\tilde{\mathbf{W}}$ represents the flip operation over both weight dimensions of the corresponding encoding parameter $\mathbf{W}$, and $\mathbf{B}'^{(l)}$ is a bias.

For downsampling layers in the encoding stage, the corresponding decoding layer corresponds to an upsampling layer.
One possible upsampling strategy for max-pooling layers  is to ``remember" the position of the input that had the maximum value for every max-pooling during the forward propagation of the input data through the network.
We will refer to a a layer that implements such a strategy as an \emph{unpooling} layer.

Given a set of training data $\mathbf{X} =\{\mathbf{x}_1, \dots, \ensuremath{\mathbf{x}}_N\}$, the parameters of a (C)AE can be optimized in an unsupervised way by minimizing the reconstruction error of $\mathbf{X}$, in a similar fashion to Eq.~\eqref{eq:parameter_learning}.
The typical loss function for (C)AEs is the mean squared error\footnote{See Eq.~\eqref{eq:MSE} in Appendix \ref{sec:cost_functions}.}.

\subsubsection{Multi-task learning}\label{sec:multitask}
Solving a problem constrained to multiple objective functions is the object of study of \emph{muticriterion optimization}~\cite{Boyd:2004uz}.
A vector optimization problem can be formalized as follows.
Given a vector loss function $\mathbf{L}(\ensuremath{\boldsymbol\theta}) = [L_1,\dots,L_q]^T$, whose components can be interpreted as $q$ different scalar objectives, to be minimized.
In order to apply standard scalar optimization methods, such as gradient descent, we can ``scalarise'' a multi-criterion problem by forming a weighted sum objective, i.e.
\begin{equation}
L = \boldsymbol\alpha^T \mathbf{L}(\ensuremath{\boldsymbol\theta}) = \sum_{i=1}^q \alpha_i L_i(\ensuremath{\boldsymbol\theta}),
\end{equation}
where $\alpha_i$ represents a weighting coefficient for the $i$-th component of $\mathbf{L}(\ensuremath{\boldsymbol\theta})$.
This weights are usually constrained to $\alpha \leq 0$ and $\sum_i^q\alpha_i = 1$ .
Using this framework, it is possible to express the multi-task objective as minimizing a joint loss function for both classification and autoencoding as follows
\begin{equation}
L(\ensuremath{\boldsymbol\theta}) = (1 - \alpha_{MT}) L_{CL}(\ensuremath{\boldsymbol\theta}) + \alpha_{MT} L_{AE}(\ensuremath{\boldsymbol\theta}),
\end{equation}
where $\alpha_{MT}$ is the multi-task weight coefficient and $L_{CL} = L_{CCE}$ and $L_{AE} = L_{MSE}$ represent the loss functions for the classification and autoencoding tasks, respectively.

\subsubsection{Transfer of a CNN model from one task to another}\label{sec:transfer-cnn-model}
Given a CNN  with $N$ layers used for classification, an CAE can be built as follows.
For layers $l_{N-1}, \dots l_{1}$, (i.e., all layers in the CNN except the classification stage $l_N$), build the mirroring decoding layer, corresponding to Eq.~\eqref{eq:decoding_fully_connected} if the encoding layer is a fully connected layer, Eq.~\eqref{eq:decoding_convolutional} if the encoding layer is a convolutional layer or an unpooling (upsampling) layer if the encoding layer is a max-pooling (subsampling) layer.
Conversely, given a CAE with $2(N -1)$ layers, a CNN classifier can be built by removing the decoding layers from the CAE (layers $l_N$ to $ l_{2(N-1)}$) and appending a fully connected layer with with as many softmax units as the desired number of classes.

In order to reduce the degrees of freedom, in the transfer of a model from one task to another, we do not use biases in the decoding layers of a CAE built from an autoencoder.
This guarantees that only the weights (present in both models) are responsible for learning the task.

\subsection{Strategies for conceptual change in CNNs}\label{sec:strat-conc-change}
In this Subsection, we propose a number of strategies for adapting learned representations in response to new tasks.
For clarity, we label each choice with a keyword, and refer to the final strategies as a combination of keywords, listed at the end of this section.

\subsubsection{Two baselines}\label{sec:two-baselines}
There are two obvious baseline approaches to the adaptation of a model to a new task.
One is to make no changes to the model upon the change of task.
In this case, the only change is that the training data of the old task is replaced by the training data of the new task.

A practical issue is that a new task implies a new interpretation of the outputs, and possibly a different number of outputs.
Reusing the parameters of the output layer for the new task raises the question of how the output variables for the old task should be mapped to those of the new task, a mapping that is necessarily arbitrary.
For this reason, when we reuse the parameters of a prior model, we replace its output layer (the rightmost layer in Figure~\ref{fig:cnn_architecture}, page~\pageref{fig:cnn_architecture}) with a new layer, initialized with random parameter values.
Ignoring this detail, we refer to this baseline strategy as REUSE ALL.

Another, contrary approach is to altogether ignore the representations learned on the prior task, and start with a randomly initialized model to learn the new task.
This can be regarded as a case of infinite plasticity, where the new task forms the representations without any trace of the representations that were learned in the prior task.
This strategy will be referred to as RESET.

\subsubsection{Selective reuse of the prior model: keep convolutional filters}\label{sec:select-reuse-conv-filt}
Considering the dynamics of learning, REUSE ALL may not be ideal, since it starts learning task 2 with a model that is specialized on task 1.
It may take more effort to ``unlearn'' aspects of task1 that are irrelevant for task 2, than to learn task 2 from scratch.

However, if the data in task 1 and task 2 are similar in nature, it is likely that the data have at least some common structure.
For example, in the case of natural images depicting different classes of objects, it is likely that certain low level representations, such as local edges in natural images, are useful for different tasks, such as the recognition of different object classes.
Different classes of objects may involve different constellations of similar forms.
For example, a dark round shape may represent wheels on a car, portholes in a ship, or the eyes of an animal.

In the current experiment, this observation inspires an adaptation strategy where the convolutional filters --- representing the low level representations --- are preserved across tasks, whereas the rest of the network is re-initialized to a random state.
This option is referred to as REUSE CF.

\subsubsection{Prior regularisation}\label{sec:prior-regularisation}

A strategy for domain adaptation proposed by \citet{chelba2006adaptation} is to take a maximum a-posteriori approach to the estimation of the model parameters for the new task, where the model learned on the first task provides a prior estimate of the parameters.
This prior serves as a regulariser for the parameters.

Standard parameter regularisation schemes are based on the assumption that the parameters \ensuremath{\boldsymbol\theta}{} follow a zero-mean gaussian distribution, that is $p(\ensuremath{\boldsymbol\theta}) \sim \mathcal{N}(\mathbf{0}, \frac{1}{\lambda} \mathbf{I})$, leading to the addition of a regularisation term $\frac{\lambda}{2} ||\ensuremath{\boldsymbol\theta}||^2_2$ to the standard expression for the loss, in Equations~(\ref{eq:MSE}) and (\ref{eq:CCE}), page~(\pageref{eq:MSE}).

The prior proposed by \citet{chelba2006adaptation} is:
\begin{equation}
\label{eq:prior}
p(\ensuremath{\boldsymbol\theta}) \sim \mathcal{N}(\ensuremath{\params_{\mathit{old}}}, \frac{1}{\lambda} \mathbf{I}),
\end{equation}
a gaussian distribution around the parameters \ensuremath{\params_{\mathit{old}}}{} that were learned on the first task.
This assumption leads to an alternative loss function $L_\mathit{PR}$, including the prior regularisation ($\mathit{PR}$) term \cite{daume07easyadapt}:

\begin{equation}
\label{eq:priorreg}
L_\mathit{PR}(\ensuremath{\boldsymbol\theta}) = L(\ensuremath{\boldsymbol\theta}) + \frac{\lambda}{2} ||\ensuremath{\params_{\mathit{old}}} - \ensuremath{\boldsymbol\theta}||^2_2.
\end{equation}

In the current experiment, the prior regularisation is applied to the convolutional filters, and used in combination with the RESET option; it is referred to as RESET PRF.

\subsubsection{A note on random initialization}\label{sec:note-rand-init}
The idea of investigating the variance of the layer outputs to improve weight initialization for deep learning was introduced in~\cite{Glorot:2010uc}.
This result has motivated the search for careful initialization, rather than unsupervised pretraining of networks with methods such as RBMs, thus representing a considerable speedup in the training of neural networks .
\citet{Glorot:2010uc} suggest  that keeping the layer-to-layer transformations such that the singular values of the Jacobian matrices associated with each layer\footnote{the Jacobian matrix associated with the $l$-th layer is given by $\mathbf{J}^{(l)}= \dpar{\mathbf{y}^{(l+1)}}{\mathbf{y}^{(l)}}$.} are approximately  1 is equivalent to keeping the ratio of the average activation variance going from layer $\mathbf{y}^{(l)}$ to layer $\mathbf{y}^{(l+1)}$.
 This result implies that the random initialization of the model parameters for each layer depends on the nonlinear activation function and the size of the layer.
The main intuition behind this initialization strategy is that if the network parameters are initialized to small, the output shrinks while passing through each layer, and eventually being to small.
On the other hand, if the network parameters are too large, the output of each layer keeps growing, until the output of the network is saturated.

We use the same random initialized parameters across adaptation strategies to rule out initialization as a noise factor for each run of the experiment.

\section{Experiment}\label{sec:experiment}
The adaptation strategies are compared on three different data sets, described in Section~\ref{sec:datasets}.
For each data set the following procedure is followed.

\begin{enumerate}
\item The data set is split into a source and a target domain, as described in Section~\ref{sec:problem-desc}.
For the specific partitioning per data set, see Section~\ref{sec:datasets}
\item A model is trained in the source domain for each of three tasks: classification, autoencoding, and the multi-task of simultaneous classification and autoencoding
\item A model is trained in the target domain for each of two tasks: classification, and autoencoding.
For each adaptation strategy, two training runs are realized
\item Each model is evaluated on the target test set during training, in order to monitor adaptation.
Per adaptation strategy, the results of the two training runs are averaged, in order to reduce the impact of random effects
\end{enumerate}

A schematic overview of the above process is given in Figure~\ref{fig:diagram}.
Since the above process produces a multitude of models, each trained under different conditions, we will refer to them using combinations of labels denoting the adaptation strategy used, and labels denoting the prior model used by the adaptation strategy.
The labels are listed in Table~\ref{tab:adapt-strategies}.

\begin{figure}[ht!]
\centering

\includegraphics[width=\linewidth]{figs/experiment-diagram}
\caption{Schematic overview of the experiment for a single data set; grey rounded boxes represent training methods, circles represent models, document shapes represent data instances, and the white rounded box represents the evaluation method}
\label{fig:diagram}
\end{figure}

\begin{table}[t]
\centering
\begin{tabular}[t]{lp{.6\linewidth}}
RESET & Initialize parameters with random values \\
RESET PRF & Prior regularisation on convolutional filters \\
REUSE ALL & Initialize all parameters (except output layer) from prior model \\
REUSE CF & Initialize only convolutional filter parameters from prior model \\
(CL) & Prior model trained as classifier \\
(AE) & Prior model trained as autoencoder \\
(MT) & Multi-task: Prior model trained simultaneously as classifier and autoencoder
\end{tabular}
\caption{Labels denoting adaptation strategies, and their meaning, as used in the results.
The labels in parentheses represent prior models, to be used in conjunction with one of the adaptation strategies}
\label{tab:adapt-strategies}
\end{table}

Since we are interested in adaptation methods that allow for a quick adaptation of a model from the source task/domain to the target task/domain, we intentionally limit the size of of the training (and validation) set in the target domain.
The scarcity of training samples makes it harder for the model to adequately generalize in the target domain, and thus it increases the potential benefit of adapting a model from another domain (although the actual benefit of course depends on the resemblance of the domains).

\subsection{Data Sets}\label{sec:datasets}
\paragraph{MNIST} \label{sec:datasets:mnist}
The Mixed National Institute of Standards and Technology (MNIST) database  consists of handwritten digits collected by American high school students and employees of the United States Census Bureau~\cite{Lecun:1998hy}.
This database constitutes one of the most used data sets for benchmarking machine learning algorithms~\cite{Bishop:2006}.
The MNIST consists of 70,000  gray-scaled images, rescaled to fit in a $20\times20$ pixel box, and then centred in a $28\times20$ box.
For these experiments, MNIST was divided into two subsets, namely Data Set 1, consisting of digits $\{0, 1, 2, 3, 4\}$, and Data Set 2, consisting of digits $\{5, 6, 7, 8, 9\}$.
The training set is divided into 50,000 examples for computing the parameter updates, 10,000 examples for validation and 10,000 examples for testing.
Data Set 1 contains 25,538 examples for training, 5,058 for validating and 5139 for testing.
For Data Set 2, 10 samples per class were randomly selected for both training and validating, making a total of 50 examples for training, 50 for validating and 4861 for testing.

\paragraph{CIFAR-10}\label{sec:datasets:cifar10}
The CIFAR-10 is a labelled subset of the \emph{80 million tiny images} data set collected by Krizhevsky, Nair and Hinton~\cite{Krizhevsky:2009tr}.
This data set has been used to evaluate the performance of algorithms in machine learning and computer vision.
CIFAR-10 consists of 60,000 $32\times 32$ colour (RGB) images divided into 10 classes, that comprising vehicles and animals.
In this paper, Dataset 1 was chosen to include all instances of classes ``airplane'', ``automobile'', ``bird'', ``cat'', and ``deer'', while Data Set 2 consists  of classes ``dog'', ``frog'', ``horse'', ``ship'', and ``truck''.
Each training set was split into training and a validation sets, consisting in $75\%$ (37,500 examples) and $25\%$ (12,500 examples) of the data, respectively, while the test set contains 10,000 examples. Data Set 1 consists of 18,681, 6,319 and 5,000 examples, respectively for training, validating and testing, while  Data Set 2 contains 10 randomly selected samples per class for training and 10 randomly selected samples per class for testing, to make to make a total of 50 training examples, 50 validating examples and 5,000 examples for testing.
\paragraph{Composers}\label{sec:datasets:composers}
As an application of the proposed methods in a musical domain, a data set consisting of excerpts of musical pieces of the baroque and classical periods was used.
These excerpts are represented as piano-rolls, i.e., images where each pixel in the y axis corresponds to a musical note (using the MIDI note number convention), and each pixel in the x axis represents a unit of time.
 The scores where taken from Muse Data database\footnote{\url{http://www.musedata.org}}, an electronic library of classical music scores created by the Center for Computer Assisted Research in the Humanities at Stanford University.
Each excerpt has a length of 50 quarter notes, with a sample rate of a 32nd note (an 8th of a quarter), with a hop size of 10 quarter notes between contiguous windows.
All pieces have been centred to fit in a MIDI range of 68 notes.
To signalize the end of a note, the its last 32nd is left blank.
Data Set 1 consists of a selection from excerpts from G.
P.
Telemann's Cantatas, J.
S.
Bach's Cantatas and G.
F.
H\"andel's Concerti Grossi and Trio Sonatas,  while Data Set 2 consists of excepts from string quartets by F.
J.
Haydn and W.
A.
Mozart.
All excerpts coming from a single piece\footnote{Movements of a work are considered individual pieces.} appear only in either the training, validation or test sets, i.e., there are no pieces that appear on more than one  data set.
There are a total of 2073 training examples, 1393 validation examples and 993 testing examples, with Data Set 1 containing 1141, 755 and 541 examples for training, validating and testing, respectively and Data Set 2 consisting of 10 randomly selected samples per class for training and 10 randomly selected samples per class for validation, to make a total of 20 examples for training, 20 examples for validating and 452 examples for testing.

\subsection{Model training}\label{sec:model_training}

All models were trained using RMSProp~\cite{Dauphin:2015wn}, a derivative of the traditional backpropagation algorithm.
This method is a mini batch variant of stochastic gradient descent that adaptively adjusts the learning rate by dividing the gradient by an average of its recent magnitude.
In order to accelerate gradient descent, we use Nesterov's method for accelerating gradient descent~\cite{Sutskever:2013uwa}.
 In order to avoid overfitting, several strategies are used, including $l_2$-norm weight regularisation, enforcing sparseness in layer activations, early stopping  and dropout.
Regularisation of the $l_2$ norm enforces sparse parameters~\cite{Bishop:2006}, while the sparsity in layer activations was enforced using Hoyer's sparseness measure~\cite{2004cs........8058H}.
Dropout prevents overfitting and provides a way of approximately combining different neural networks efficiently by randomly removing units in the network, along with all its incoming and outgoing connections~\cite{Srivastava:2014ww,Hinton:2012wr}.
The network architectures and hyper-parameters were empirically selected by optimizing the models to their respective validation sets.

\paragraph{MNIST}\label{sec:training:mnist} The classifier consists of a CNN with a convolutional layer with 32 kernels of size $5 \times 5$ and sigmoid activations ($l_1$), followed by a max-pooling layer with a pool size of $2\times2$ ($l_2$).
The convolutional stage is followed by a classification stage consisting of a fully connected layer with 40 sigmoid units and ($l_3$) and a fully connected layer with 10 softmax units.
Dropout is used after the convolutional stage, i.e., after $l_2$.

The learning rate for RMSProp is set to $10^{-5}$, Nesterov's momentum to 0.5, the probability of dropout is set to 0.5, the regularisation coefficient is $0.001$, the sparsity coefficient is set to $10^{-4}$, the target sparseness is $0.9$ and the batch size is 50.
 The network was trained for a maximum of 2000 epochs, with a maximum of 200 epochs from the best result for early stopping.
The multi-criterion weighting coefficient is set to $0.01$ in multi-task models.

\paragraph{CIFAR-10} The classifier consists of a CNN with a convolutional layer with 32 kernels of size $5 \times 5$ and sigmoid activations ($l_1$), followed by a max-pooling layer with a pool size of $2\times2$ ($l_2$).
The classification stage consists of a fully connected layer with 40 sigmoid units and ($l_3$) followed by a fully connected layer with 10 softmax units. As in the previous case, dropout is used after the convolutional stage, i.e., after $l_2$.

The learning rate for RMSProp is set to $10^{-5}$, Nesterov's momentum to 0.5, the probability of dropout is set to 0.5, the regularisation coefficient is $0.001$, the sparsity coefficient is set to $10^{-4}$, the target sparseness is $0.9$ and the batch size is 50.
 The network was trained for a maximum of 3000 epochs, with a maximum of 200 epochs from the best result for early stopping.
In multi-task models, the multi-criterion weighting coefficient is set to $0.01$.

\paragraph{Composers}  The classifier consists of a CNN with a convolutional layer with 9 kernels of size $9 \times 9$ and rectified linear activations ($l_1$), followed by a max-pooling layer with a pool size of $2\times2$ ($l_2$), then a second convolutional layer with 5 kernels of size $5\times 5$ with rectified linear activations ($l_3$), followed by a max-pooling layer with a pool size of $2\times 2$ ($l_4$).
The convolutional stage is followed by a fully connected layer with 256 rectified linear units and ($l_5$) and fully connected layer with 4 softmax units ($l_6$), both conforming the classification stage. In a similar fashion as in the previous cases, dropout is used after the convolutional stage, i.e., after $l_4$.

The learning rate for RMSProp is set to $10^{-6}$, Nesterov's momentum to 0.5, the probability of dropout is set to 0.5, the regularisation coefficient is $0.001$, the sparsity coefficient is set to $10^{-4}$, the target sparseness is $0.5$ and the batch size is 50.
 All networks was trained for a maximum of 2000 epochs, with a maximum of 200 epochs from the best result for early stopping.
For the models solving the joint classifier and autoencoding task, the multi-criterion weighting coefficient is set to $0.01$.

\section{Results and discussion}
\label{sec:resanddisc}
The results of the various adaptation scenarios and data sets are displayed in Figures~\ref{fig:acc},~\ref{fig:clloss}, and ~\ref{fig:aeloss}.
Figure~\ref{fig:acc} shows the classification accuracies for the classification task $\ensuremath{\mathcal{T}}_t^{CL}$ in the target domain $\ensuremath{\mathcal{D}}_t$, for each of the three data sets.
The fact that the accuracy curves are rather noisy is because the classification accuracy is not the training objective, but rather the categorical cross-entropy of the model output with the one-hot representation of the class labels (plotted in Figure~\ref{fig:clloss}).
Furthermore, the occasional discontinuities in the curves are due to the averaging over multiple runs, where some runs converge (and thus halt) after fewer epochs than others.

The first baseline strategy (RESET) consists in a random re-initialisation of all parameters (conform Section~\ref{sec:note-rand-init}), meaning that no knowledge from the source task $\ensuremath{\mathcal{T}}_s$ in the source domain $\ensuremath{\mathcal{D}}_t$, is used at all.
The second baseline strategy (REUSE ALL), is to initialize the model with the parameters of the model trained on $\ensuremath{\mathcal{T}}_s$ in the source domain $\ensuremath{\mathcal{D}}_t$.
Note that there are three source tasks (CL, AE, and MT).

In Figure~\ref{fig:clloss}, there is a consistent trend that the \mbox{REUSE ALL} baseline strategy is slow to adapt, independent of the source task.
Interestingly, among the \mbox{REUSE ALL} conditions (dotted lines), the adaptation of the autoencoder model of the source domain---\mbox{REUSE ALL (AE)}---is more beneficial to learning the target classification task than the adaptation of the classifier---\mbox{REUSE ALL (CL)}---from the source domain.
This may be an indication that the autoencoder learns representations that are useful for encoding the data in general, whereas the classifier learns more specialised features that are useful primarily for recognising the specific classes that happen to be in the source domain.
To some degree, this finding underlines the rationale for \emph{unsupervised pre-training}~\cite{Erhan:2010wd}, that learning to encode the data independent of any classification objective provides more robust representations (for subsequent classifcation) than driving the representations by a classification objective (with different class labels) from the start.
The fact that the unsupervised-pretraining---\mbox{REUSE ALL (AE)}---does not surpass the \mbox{RESET} strategy suggests that the distribution of the data that the autoencoder has seen is not sufficiently representative for the target domain, i.e.
$P_s(X) \neq P_t(X)$.

The low loss values of \mbox{REUSE ALL (AE)} with respect to \mbox{REUSE ALL (CL)} and \mbox{REUSE ALL (MT)} does translate into higher accuracy rates (Figure~\ref{fig:acc}) for the CIFAR and Composers data sets, but surprisingly, it does not for the MNIST data set.

The \mbox{REUSE CF} strategy (Figure~\ref{fig:clloss}, dashed lines), which retains the first layer of convolutional filters learnt from the source domain, but resets all other parameters of the model, considerably speeds up adaptation to the target task with respect to the \mbox{RESET} baseline.
With this strategy, the adaptation from the classification task in the source domain is generally most successful, but even when adapting models from the AE and MT source tasks, the adaptation is mostly quicker than \mbox{RESET}.
A plausible explanation for this is that the low level structure of the data is usually more general, whereas the higher level structure (say, the configuration of lower level features into shapes) is more specific to particular data categories.

For the autoencoding task $\ensuremath{\mathcal{T}}_t^{AE}$ in the target domain $\ensuremath{\mathcal{D}}_t$ (Figure~\ref{fig:aeloss}), the \mbox{REUSE ALL (AE)} strategy leads to substantially better autoencoding results from the start in the target domain, but as opposed to the other adaptation strategies, the training of the model using this strategy does hardly improve the autoencoding loss beyond its initial value.
The \mbox{REUSE ALL (CL)} strategy on the other hand, adapts much worse on than the \mbox{RESET} baseline on CIFAR and MNIST, on a par with \mbox{REUSE CF (CL)}.

The autoencoder task in the target domain of the Composers data set shows a very different pattern (Figure~\ref{fig:aeloss}, right).
Here, all adaptation strategies outperform the \mbox{RESET} baseline that discards any knowledge from the source domain.
This may be an indication that the source and target domains in the Composers data set have rather similar marginal distributions, i.e., $P_s(X) \approx P_t(X)$, giving the adaptation strategies an advantage over the \mbox{RESET} baseline.
An explanation for this may be that there is a lot of structure in the musical data that is common across the music of the different composers (tonal structure, rhythmical structure).
The prevalence of this common structure would also explain the rather low overall accuracy in the composer classification task (Figure~\ref{fig:acc}, right).

The simultaneous learning of both autoencoding and classification in the source domain appears to provide more useful knowledge to be transferred to the autoencoding target task (Figure~\ref{fig:aeloss}), than the classification target task (Figures~\ref{fig:acc}, and ~\ref{fig:clloss}).
In the latter case, adaptation is more successful when the source task is also classification.

In the prior regularization strategy (\mbox{RESET PRF}), the first layer convolutional filters of a randomly initialised network are biased towards the filters learned from the source domain and task.
This regularization does improve a bit upon the \mbox{RESET} baseline sometimes, but not very consistently, and the gains are usually moderate.

One issue that we have not discussed is whether the asymptotic performance of some strategies is substantially different from that of others.
From the Figures it is clear that broadly speaking, the adaptation strategies converge to a certain range, even if there are individual differences.
To make better judgements on the asymptotic performance of the adaptation strategies however, it is necessary to do more extensive experimentation, with longer training phases, more data sets, and averaging results from more runs of the same condition.
Moreover, although the hyperparameters of the models (e.g.
the number of convolutional filters, the pooling size, regularisation constants) were selected through exploration prior to the experiment, a more exhaustive study of the hyperparameters may provide a better view on the value of the adaptation strategies.
From a computational point of view, this is a considerable undertaking, given that the results presented here require several weeks of continuous computation on multiple GPGPU-enabled machines.

\begin{landscape}\centering
\begin{figure}
\includegraphics[height=1.15\textheight]{figs/results/CIFAR_test_accuracy_CL}
\includegraphics[height=1.15\textheight]{figs/results/MNIST_test_accuracy_CL}
\includegraphics[height=1.15\textheight]{figs/results/Composers_test_accuracy_CL}
\caption{Classification accuracy (proportion of correctly classified instances) on the target test-set of the different data sets, for different adaptation strategies.
Curves are averaged over two runs.
See table~\ref{tab:adapt-strategies} (page~\pageref{tab:adapt-strategies}) for an explanation of the legend}
\label{fig:acc}
\end{figure}
\end{landscape}

\begin{landscape}\centering
\begin{figure}
\includegraphics[height=1.15\textheight]{figs/results/CIFAR_test_loss_CL}
\includegraphics[height=1.15\textheight]{figs/results/MNIST_test_loss_CL}
\includegraphics[height=1.15\textheight]{figs/results/Composers_test_loss_CL}
\caption{Classification loss (categorical cross-entropy) on the target test-set of the different data sets, for different adaptation strategies.
Curves are averaged over two runs.
See table~\ref{tab:adapt-strategies} (page~\pageref{tab:adapt-strategies}) for an explanation of the legend}
\label{fig:clloss}
\end{figure}
\end{landscape}

\begin{landscape}\centering
\begin{figure}
\includegraphics[height=1.15\textheight]{figs/results/CIFAR_test_loss_AE}
\includegraphics[height=1.15\textheight]{figs/results/MNIST_test_loss_AE}
\includegraphics[height=1.15\textheight]{figs/results/Composers_test_loss_AE}
\caption{Autoencoding loss (mean squared error) on the target test-set of the different data sets, for different adaptation strategies.
Curves are averaged over two runs.
See table~\ref{tab:adapt-strategies} (page~\pageref{tab:adapt-strategies}) for an explanation of the legend}
\label{fig:aeloss}
\end{figure}
\end{landscape}

\section{Conclusions}\label{sec:conclusions}
When interacting with a dynamical and unpredictable environment, the ability of an agent to easily adapt the conceptual space with which it interprets the environment, is a strong advantage.
As argued in Section~\ref{sec:from-adapt-repr}, this adaptability may also be one of the underlying mechanisms of creative behaviour.
In this document, we have described and evaluated several adaptation strategies to transform learned representations optimized for a specific task in a specific domain (the \emph{source} task/domain), to another task in another domain (the \emph{target} task/domain).
We have evaluated the strategies using a convolutional neural network, on multiple data sets, comprising natural images, handwritten digits, and classical music.

The results show that, across domains and tasks, adaptation strategies that transform existing representations allow for a quicker adaptation to a new task in a new domain than starting the representation learning from scratch.
Although there is no single adapatation strategy that is universally superior to others, some clear patterns do emerge from the results.
Firstly, when the target task is classification, then a successful adaptation strategy is to keep the first level convolutional filters (i.e.
the lower level representations) from the source task/domain, and reset the rest of the parameters.
This strategy is even beneficial when the source task is autoencoding, rather than classification.
Secondly, for the autencoding target task, the reuse of the full model (rather than just the convolutional filters) is a successful strategy, in the sense that even from the start, the task performance with that strategy is comparable with the asymptotic performance of other strategies.
But in contrast to the other strategies, this strategy does not substantially improve the model beyond the initial performance.
The experiments do not provide evidence for a strong advantage of the prior regularization of the convolutional filters (Section~\ref{sec:prior-regularisation}), over the condition where the model is trained from scratch with only standard regularization.

More elaborate experiments are required to provide firmer conclusions on the merits of each of the adaptation strategies.
In particular, a longer training phase is necessary to provide better insight in the asymptotic behaviour of the strategies.

\section*{Acknowledgement}\label{sec:acknowledgement}
The project Lrn2Cre8 acknowledges the financial support of the Future and Emerging Technologies (FET)
programme within the Seventh Framework Programme for Research of the European Commission, under FET
grant number 610859.

\bibliographystyle{apalike}

redpajama_set_name: RedPajamaArXiv


text:
Transportation infrastructures
Underground and railway system
Port and Maritime Engineering
Water, Sanitation and Waste
Sustainability and Energy
Construction Management and Supervision
GEOLAB
Investment and Development Agency
English PT
Homepage > About Us > Ethics and compliance
FUTURE Compliance Programme aims to promote a culture of integrity and transparency based on the Group's values.
At FUTURE Group we strive to be on the right direction.
FUTURE Compliance Programme aims to promote a culture of integrity and transparency based on the Group's values, its Code of Moral and Ethical Conduct, and legal determinations. This is ensured through policies and standards and a set of corporate procedures that guarantee the effectiveness and efficiency of the Group's compliance programmes in identifying, preventing, detecting and correcting non-compliance with applicable laws and regulations.
It is therefore the responsibility of FUTURE Group:
To attest the compliance of the company and its employees with the procedures and policies of ethics and compliance in force
To communicate policies and train all Group employees in the approved procedures, ensuring that they are well understood and respected
To monitor compliance with approved policies and procedures by measuring and assessing the level of compliance throughout the organization
To investigate any incident or non-compliance with legal requirements or standards in force within the Group
To ensure instant communication on any non-compliance and relevant compliance facts
To ensure the independence of its operation in matters related to compliance
Future Group Headquarters
Future Manifesto
Terms and conditions Cookies Policy Privacy Policy Career Opportunities
FUTURE © 2021 - All rights reserved

redpajama_set_name: RedPajamaCommonCrawl


Fresh Every Thyme is a personal chef business serving clients in the Seattle and greater King County area. We handle all the menu planning, shopping, and cooking. We'll deliver fresh meals, ready to heat and serve, to your home.
A personal chef is a great resource for people who want to eat well but who don't like or have time to cook.
Take back your time and let Fresh Every Thyme take over your meal prep!
We prepare a week's worth of meals for you, made to fit your tastes and dietary requirements, and deliver everything to your home. You simply heat and serve.
We take care to cook with fresh and high quality ingredients and personalize the menu for you.
We will work with you to cook and grocery shop for your upcoming visit and we'll arrange to have your meals and groceries in the refrigerator before you arrive.
Let us help get the meals that fit your tastes prepared so you don't have to worry about preparing food.
We offer catering services, from full service events to drop off or pick up. We'll work with you to create a menu that fits your desired guest experience and your budget.
We cater business lunches too.
Hello and thank you for visiting our site. I am the chef/owner of Fresh Every Thyme. I am lucky enough to be able to take my passsion of cooking for family and friends and build it into this business. I now get to cook for many different clients and I'm loving every minute of it.
I love planning menus, cooking new dishes, and most of all cookig for others. Cooking is a way I can give back.
Over the years I have honed my skills at finding dishes that match my client's unique dietary needs. I take each new diet as a challenge to find the right dishes to meet the client's taste profile.
I look forward to cooking for you soon!


text:
Q: Horizontally center small triangle at bottom of div and resize responsively and proportionately With CSS I am trying to accomplish the following effect that responds to screen resizing (for responsive design). Specifically I want the triangle to get proportionately smaller as the browser window gets smaller. I have mocked this up as a PNG (It's not HTML/CSS):

So far I am using this code as a base, with 2 divs stack on top of one another:
CSS:
.pinkbox {
    background-color: #FF8290;
    width: 100%;
    height: 100px;
}

.greenbox {
    background-color: #85D782;
    width: 100%;
    height: 100px;
}

HTML:

A: Responsive triangles with PURE CSS and without media queries.
See this codeitdown article
FIDDLE
Resize the window and watch the triangle resize responsively !
Markup

CSS
.top
{
    background: pink;
    height: 100px;
    position: relative;
}

.bottom
{
    background: lightGreen;
    height: 100px;
}
.triangle-down{
    width: 2.5%;
    height: 0;
    padding-left:2.5%;
    padding-top: 2.5%;
    overflow: hidden;
    position: absolute;
    left:0;right:0;  /* center the arrow */
    margin:auto;  /* center the arrow */
    top: 100px; /* height of top section */
    z-index:1;
}
.triangle-down:before {
    content: '';
    display: block;
    width: 0;
    height: 0;
    margin-left:-50px;
    margin-top:-50px;

    border-left: 50px solid transparent;
    border-right: 50px solid transparent;
    border-top: 50px solid pink;
}

How the heck does it work?
Well, firstly we need to decide two things:
1) The width/height ratio of the triangle. In the above example (and for simplicity) I used a ratio of 2:1.
2) How much of the container/viewport width we want our triangle to take up. In the above example I used a triangle of 5% of the viewport width: (width 2.5% + padding-left:2.5%;)
Now set up the other properties /proportions according to the following rules: (from above article)

1) (padding-left + width)/padding-top = (border-left +
  border-right)/border-top = base/height
2) margin-left = -border-left = -border-right
3) margin-top = -border-top
4) width = padding-left

Customizing the responsive triangle:
Let's say you wanted a triangle of ratio 3:1 and for it to take up 6% of the width..
No problem!
ANOTHER FIDDLE (This one actually looks more like the picture)
Modify CSS to this:
.triangle-down{
    width: 3%;
    height: 0;
    padding-left:3%;
    padding-top: 2%;
    overflow: hidden;
    position: absolute;
    left:0;right:0;
    margin:auto;
    top: 100px;
    z-index:1;
}
.triangle-down:before {
    content: '';
    display: block;
    width: 0;
    height: 0;
    margin-left:-50px;
    margin-top:-33px;

    border-left: 50px solid transparent;
    border-right: 50px solid transparent;
    border-top: 33px solid pink;
}

Enjoy!

A: It is not possible to have a css triangle with a responsive width. It is possible if you use a background image to create the triangle.
Here is code for a css triangle with a fixed width:
CSS
.pinkbox {
    background-color: #FF8290;
    width: 100%;
    height: 100px;
    position: relative;
}
.pinkbox:after {
    content: " ";
    position: absolute;
    bottom: -20px;
    left: 50%;
    margin-left: -20px;
    border-style: solid;
    border-width: 20px 20px 0 20px;
    border-color: #FF8290 transparent transparent transparent;
}
.greenbox {
    background-color: #85D782;
    width: 100%;
    height: 100px;
}

Demo
You could add some media queries to change the triangle at certain screen widths like this:
CSS
.pinkbox:after {
    content: " ";
    position: absolute;
    left: 50%;
    bottom: -20px;
    margin-left: -40px;
    border-style: solid;
    border-width: 20px 40px 0;
    border-color: #FF8290 transparent transparent transparent;
}
@media (max-width:640px) {
    .pinkbox:after {
        margin-left: -30px;
        border-width: 20px 30px 0;
    }
}
@media (max-width:480px) {
    .pinkbox:after {
        bottom: -15px;
        margin-left: -20px;
        border-width: 15px 20px 0;
    }
}

Demo
Or you could add some jQuery to change the width of the triangle on window resize:
Javascript
$(window).on('resize', function () {
    arrowWidth = $(this).width() / 20;
    $('.pinkbox .arrow').css({'margin-left' : - arrowWidth, 'border-width' : '20px ' + arrowWidth + 'px 0'});
}).trigger('resize');

Demo

A: You can certainly change the size of a css triangle based on window width with @media css
@media (max-width:960px) {

.pinkbox:after {
    content: " ";
    position: absolute;
    bottom: -20px;
    left: 50%;
    margin-left: -20px;
    border-style: solid;
    border-width: 20px 20px 0 20px;
    border-color: #FF8290 transparent transparent transparent;
    }

}

@media (max-width:700px) {

.pinkbox:after {
    content: " ";
    position: absolute;
    bottom: -15px;
    left: 50%;
    margin-left: -15px;
    border-style: solid;
    border-width: 15px 15px 0 15px;
    border-color: #FF8290 transparent transparent transparent;
    }

}

@media (max-width:500px) {

.pinkbox:after {
    content: " ";
    position: absolute;
    bottom: -10px;
    left: 50%;
    margin-left: -10px;
    border-style: solid;
    border-width: 10px 10px 0 10px;
    border-color: #FF8290 transparent transparent transparent;
    }

}

A: Non Responsive, but 100% accurate on center.
http://jsfiddle.net/SinisterSystems/Z8v97/1/

EDIT - Working on responsive solution. One moment please.

redpajama_set_name: RedPajamaStackExchange


text:
Killings in Southern Kaduna must stop – MSSN
Moshood Adebayo
The Lagos State branch of the Muslim Students' Society of Nigeria (MSSN), has lamented the incessant killings in the Southern Kaduna, just as he called for a stop to what he described as callous killings of Nigerians.
MSSN President, Miftahudeen Thanni, who made the plea during a press conference at the Bagauda Kaltho Press Centre today, also urged government to check the perpetrators of what he described as evils going on in the country. and bring them to records.
"One of the primary responsibilities of government is to secure the citizens and the properties of the citizens. Security is key, so because of this we still implore the government to do the needful, the government is trying but they have not done their best, if they can do better i think this will be a forgotten issue.
"I also want to implore citizens and indigenes of Kaduna state especially Southern Kaduna in particular to sue for peace, to embrace peace at all time, not only by speech but by all their activities.
"They should exhibit peace at all time with each other, they should be tolerant at all time", adding that if they can do all this, there will be an enduring peace in Kaduna at all time.
Thanni also called on both the Academic Staff Union of Universities (ASUU) and the Federal Government to reach a consensus on the controversy surrounding the Integrated Payroll and Personnel Information System (IPPIS).
The MSSN president, lamented the controversy, which he described as saddening and weakens the hope in the reformation of the education sector.
"It will be unfair for the resolvable difference to affect the resumption of millions of Nigerian students at the end of the coronavirus lockdown," he said.
"The government must understand that massive investment in education is a guarantee for achievements in economic development and the well-being of their citizens."
The society further decried the poor funding of education in the country, saying it is having negative effect on output across all levels.
Edo factional Assembly meets, sets up 17 Committees
Makinde orders re-arrest of suspected serial killer in Oyo
Edo guber: Ewohinmi residents vow to deliver Obaseki


Hey buddy, stop using Microsoft software. This is like buying McDonald's and complaining that it isn't good food.
Yup..Cause Microsoft is going to do what they damn well pleases. They always have and they always will..Ok,except for that incident with the ghastly desktop.


I LOVE chickens and I have ZERO concept of personal space!
All of mom's leg bruises can be attributed to Moose not understanding personal space.
Moose is great at "moosing" around in the wild, running through bush and forest looking for anything that moves.
Moose is a lover, he loves everyone and everything. He will definitely come help you when you're trying to tie your shows, he will trample you with his love anytime you see him, he will walk all over you as if you're part of the couch. He just has so much love to share that he's not afraid to show it to you by being very, very, very close to your face at all times possible!
He also loves his pet chickens, and by love I mean he lunged at one of them when we tried to introduce them, and poor chickie is missing some tail feathers. He stalks them in their coop all day long, his bird-dog instincts are out in full force, always.


If your organization follows the financial periods to track the labor hours from project and non project work, you should define financial periods. Define financial periods to match your organization�s fiscal calendar.
The system creates Financial Periods according to specifications entered above, and the periods displayed as in Fiscal Periods page.
If you would like to start/end date of the first fiscal period, you can edit the date by clicking the cell. Date Picker will be displayed to select the desired date. Once you select the date and click �Save�, the following period dates will be adjusted automatically.
If you want to redefine the fiscal period further because of a mistake you made, you need to first select the period, and then delete it by clicking Delete, and then redefine it by clicking Define.


text:
Gel's a good idea, but so is prevention
By Kathy Sena
It doesn't make sense to stock up on fire-retardant gel and then skip fire-prevention basics, said Karen Terrill, spokeswoman for the California Department of Forestry and Fire Protection in Sacramento.
Although the department supports homeowners' use of fire-retardant gels, "it's also important for people to obey the law," she added, regarding clearing brush and removing items that can provide fuel for a fire.
According to California Public Resources Code 4291, homeowners with property adjoining mountainous areas, forest-covered lands or any areas covered in flammable material must:
* Clear brush and weeds a minimum of 30 feet from the home. (A 100-foot clearance is recommended.)
* Remove flammable material, such as leaves and pine needles, from the roof.
* Trim tree branches to keep them a minimum of 10 feet from chimneys. (Trimming to keep branches a minimum of 10 feet from the roof itself is also recommended.)
* Install a spark arrester (a half-inch wire-mesh screen) on all chimneys.
Fines for code violations can equal $270 per offense, according to the Superior Court of California.
The entire text for Code 4291 is available at www.fire.ca.gov/php/educationpubliccode4291.php.
In addition, firewood, cans containing flammable liquid and any other flammable items should be removed from the area surrounding the house, according to Terrill.
For homes on slopes, homeowners should clear a larger area than the minimum required by law. "Fire burns faster uphill," she said. "Just light a match and turn it upside down, and you'll see what I mean."
Trimming tree branches so that you can walk under them helps keep branches from becoming "ladder fuel" in the case of a grass fire.
"We have about 6,000 grass fires each year in California," Terrill said. "The key is to deny the fire fuel."
Sometimes homeowners with the best intentions can accidentally cause the problems they're trying to prevent.
"When you live in the city, you can mow the grass any time of day," Terrill said.
But in a wilderness area, all mowing and trimming should be done early in the day, when morning dew helps make potentially flammable materials less likely to catch fire.
"When you're mowing or using an edger, it's easy to chip a rock and throw a spark into a wild area," she said.
-- Kathy Sena


Rey Mysterio: "I Would Definitely Love To Close Out My Career With WWE"
Rey would love to come back and end it there.
Former three-time WWE World Champion Rey Mysterio has been all over the wrestling scene over the past year as he made a surprise appearance in the 2018 Royal Rumble match, worked WWE's 'Greatest Royal Rumble' PPV, made and is still making appearances for NJPW and recently competed at 'Dominion' this past June. Rey is also set to compete during the G1 Climax 28 against an unannounced opponent. "The Master Of The 619" is booked for 'All In' on September 1st and on top of that he has been working various shows on the independent wrestling scene.
Mysterio has been keeping himself busy but in interviews he never shies away from stating that he would like to return to WWE and close out his storied career there. Mysterio was present at Comic Con in his hometown of San Diego recently and was interviewed by Noelle Foley of Ringside Collectibles and stated that he would love to return to WWE to close out his 29-plus-year career.
Rey Mysterio has won every title there is to win in WWE except for the Universal and United States Championships and he commented on the possibility of him returning to the company and capturing those titles so he can become a Grand Slam Champion.
"I think any title that has been out of my hands or out of my possession at one point in my career, its definitely something that I would strive for so, yes, why not, and also a World Heavyweight Title if I ever came back to WWE would be awesome as well." He said.
As mentioned, Rey Mysterio is set for 'All In' and to view the talent roster that has been announced for the September 1st event thus far, click here. If you are going to use any of the quotes above, please credit Ringside Collectibles with an H/T Fightful for the transcriptions.


Perception is everything, and a toll free number will give your customers a good impression. Whether you're building a brand or supporting a contact center to support an existing product/service, having the right service is key to success.
Third Spectrum partners with multiple Tier1 providers to allow least cost routing on inbound toll free. We put our carriers face-to-face to bid for your business and the proof is in our pricing!
If you're paying more than $0.02 per minute on inbound toll free, give us a call today!
Control costs and geography by choosing United States, Canada, Caribbean.
Choose up to five major Tier-1 carriers to route your inbound toll free calls.
Our rate decks are updated weekly and optimize your inbound traffic to the cheapest carrier.
Multiple call delivery methods (SIP/PSTN) with robust fail over options to ensure your calls are always delivered.
talk to a toll free expert today!


우리는 중국에서 금속 스탬핑 디디 제조 업체 및 공급 업체 / 공장 전문화되어 있습니다. 금속 스탬핑 디디 중 하나 인 Timeplex Industrial Limited 중국 유명 브랜드 중 하나 인 저렴한 가격 / 저렴한 가격으로 고품질의 금속 스탬핑 디디 도매업.
Wholesale 금속 스탬핑 디디 from China, Need to find cheap 금속 스탬핑 디디 as low price but leading manufacturers. Just find high-quality brands on 금속 스탬핑 디디 produce factory, You can also feedback about what you want, start saving and explore our 금속 스탬핑 디디, We'll reply you in fastest.


If you hold over in a space, don't assume that the expired or terminated lease's terms will no longer apply to you. Rather, if your lease has a typical holdover clause, you should expect that those terms will apply during the holdover period, and that you must continue performing your lease obligations.
A typical holdover clause says that if you continue occupying your space after the lease ends, you'll become a tenant "at will" or a month-to-month tenant, subject to all of the terms, covenants, and conditions of the lease. So if you're a holdover tenant and you don't perform your obligations under the lease—such as making required repairs and paying holdover rent and CAM costs/operating expenses—you could suffer serious consequences.
While owners discourage tenants from holding over a space by including holdover provisions in their leases that penalize tenants for failing to move out according to their lease terms, you can still protect your own interests by negotiating modifications to your owner's holdover provisions. For negotiating tips, see "Make Five Modifications to Owner's Holdover Clause," in the May 2010 issue of the Insider, available in our online Archive.


text:
The Terminator Re-Forms
November 30, 2010 Michael Ashton Leave a comment
Blam! Blam! The EU fires and the T-1000 Terminator is blown to bits. Or…is it? There was the ECB's liquidity provisions of 2009, but Greece happened anyway. The EU responded to the Greek crisis with a "shock and awe" package, but in only a few months the T-1000 pulled itself together and kept coming. Now the Union is trying to build a firebreak after Ireland, a line in the sand, and finish off the crisis for good.
It doesn't look hopeful at this point. Although the bleeding among periphery bond markets slowed today, with Portugal and Ireland rallying 16 and 6bps, respectively, and Greece, Italy, and Spain selling off a mere 3-4bps, the Euro itself continued to take a beating and is now back at mid-September levels, down from $1.42 less than a month ago to $1.30 now. And there have been some tentative signs of pressures in the funding markets: front Eurodollar contracts were down 2-4bps while red and green packs were up around 6bps today. Front March is down 15bps over the last week. Since it seems very unlikely that the Fed is going to be tightening any time soon – and indeed, the Fed Funds strip hasn't moved – this seems to be a miniature version of what we saw at times during the 2008-2009 financial crisis. The state of interbank lending needs to be carefully monitored, and surely the ECB and Federal Reserve will be more proactive this time around. But it is a still a sign that all is not well.
The concern all along was that the next financial blow would come before the global economy was fully healed and it appears this concern may be validated by events. Output isn't in the sorry state it was in the middle of the Lehman debacle, but it can hardly be described as robust and healthy. The Case-Shiller Home Price Index has flattened out, and underperformed expectations today. This is not necessarily a bad thing, if properties that were formerly stranded (because their owners wanted to wait for higher prices) are trickling onto the market, but it is a reminder that we are still unwinding the last bubble. The Chicago Purchasing Managers' Report (62.5 vs expectations for 59.9) and Consumer Confidence (54.1 vs expectations for 53.0) were both incrementally stronger-than-expected, but neither has left the range of the last year. Moreover, the Jobs Hard To Get subindex of the Consumer Confidence survey actually rose slightly to 46.5.
None of this spells a collapsing economy (we might also say that it doesn't describe an economy that, on that evidence, needs a central bank to print money), but it also doesn't give the impression of obdurate Schwartzeneggerian implacability. So who's going to face the T-1000? Ben Bernanke? Jean-Claude Trichet? Jean-Claude Van Damme, maybe. Perhaps we're going about this crisis thing all wrong. Muscles from Brussels, indeed.
In the absence of Van Damme, investors are seeking protection in dollars, to a lesser extent in commodities – note that in the chart below, the dollar and the SP-GSCI over the last week have both been gaining although they typically mirror each other – and in options.
Commodities and the dollar - recently moving together, though this is hardly without precedent.
Interestingly, and perhaps because the threat is to sovereign issuers, U.S. Treasuries haven't been doing decidedly well. 10-year yields remain 40bps off the lows set at 2.40% in October. Inflation-linked bonds have also been going nowhere fast, although weakening a smidge at the margin. We know the story about equities: somehow treading water and bouncing again today off the 1174-75 level in the S&P. But the VIX is back to levels last seen in early October, although a far cry from the crisis spikes of 2001, 2002, 2007, 2008, and 2010 (May).
Between the VIX and the whiff of LIBOR funding pressures, I suspect that equities might be vulnerable since the momentum of last month has now firmly evaporated. It is courageous to sell stocks into year-end these days, but if the S&P breaks below that 1174ish level there is a good deal of space on the downside before useful support.
Bulls will cross their fingers for the few days leading up to Employment. Tomorrow, the ADP (Consensus: +70k vs +43k last) provides an early snapshot. The ADP report isn't a great look at Employment, especially when the government is taking a very heavy role in hiring, but it is the best indicator we have. And yet, over the last year the Payrolls number has run 876k higher in aggregate than the ADP figure. Coming into 2010, the 5-year average of rolling 12-month periods was about 96k per year, so a very large gap has opened up between Payrolls and ADP. Except for periods when Census firings were dampening Payrolls, ADP hasn't been above Payrolls for a year. Consensus estimates are institutionalizing this expectation, with economists looking for +70k on ADP versus +145 for Payrolls.
I think it is reasonable to suppose that either ADP needs to start catching up to Payrolls, or Payrolls needs to start sagging towards ADP. I suppose I think the latter is slightly more likely since the surveys of consumer attitudes suggests a weaker Employment picture rather than a stronger one, but from a trading standpoint the more relevant observation is that ADP has upside risk and Payrolls downside risk as long as economists forecast – reflexively, it seems to me – the wide spread to continue.
December 1st also brings the final revisions to Q3 Productivity and Unit Labor Costs data, and (more importantly) the ISM Manufacturing report (Consensus: 56.5 from 56.9). The Beige Book is also due out tomorrow afternoon, and car sales throughout the day.
Categories: Economy, Employment


Institute of Cancer Research Repository - ROCK and Dia have opposing effects on adherens junctions downstream of Rho.
ROCK and Dia have opposing effects on adherens junctions downstream of Rho.
ROCK and Dia have opposing effects on adherens junctions downstream of Rho. Adherens junctions (AJs) are crucial for maintaining the integrity of epithelial tissues and are often disrupted during tumour progression. Rho family proteins have been shown to regulate adherens junctions. We find that activation of the effector kinase ROCK and acto-myosin contraction disrupts AJs downstream of Rho. In contrast, signalling through the Rho effector Dia1 is required to ensure a dynamically stable interface between cells and the maintenance of adherens junction complexes. The ability of Dia1 to regulate the actin network is crucial for the localization of adherens junction components to the cell periphery.


text:
Don Lemon confronted by racist Trump supporters while filming 'I can't wait for CNN to fire your Black ass, you fa**ot'
CNN anchor Don Lemon has been feeling the hate in the Trump era and revealed on Thursday the extent of harassment and threats he's received from supporters of the president.
—The Central Park Five: 5 attorneys explain why the prosecution's disgraceful case should be reviewed—
Lemon said during an industry conference:
"I was doing a shoot in the park the other day and someone said, 'We built this country. I can't wait for CNN to fire your black ass, you faggot,'" Lemon told the audience, according to The Hollywood Reporter.
Lemon said he's been bombarded with hateful rhetoric and receives at least 300 calls a day from a Trump supporters who hate him. People have also gotten hold of his address and sends him loads of hate mail too.
The attacks, he said, "take a toll on me, on my privacy and safety and security, and the safety and security of my family."
He's also been the victim of having the cops called on him for nothing, which has become a phenomenon for those in the Black community who have endured white people making false 911 calls.
"People call the authorities and accuse me of doing certain things," Lemon said.
Lemon said he's trying to figure out how much longer he's be able to tolerate a toxic culture.
"I wonder how long I will continue to do this particular job in the way I do it."
—Spike Lee calls for Hollywood to 'shut it down' in Georgia over abortion ban—
But he admits he's in it for the long haul.
"It ain't easy, but I don't plan to give up."


The Fargo 82600 polyguard 0.6mm clear overlaminate is a reliable choice for adding a layer of high-quality protection to your cards. It guarantees to increase your cards lifespan by effectively shielding it from daily wear and tear. Each roll can yield up to 250 outstanding print results.


text:
Q: How to make installed .exe show proper details with Inno Setup under Windows I use Inno Setup to create a Windows installable .exe for a Java app. I have  properly defined AppName, AppVersion and others in the .iss file. The info appear in the installable .exe details.
The problem is that after installation, the new executable .exe in the app folder doesn't have the same details, but takes JDK details. JDK has been packaged as a runtime dependency as well. File Description and Product Name appear the same as those of JRE.

A: Look at the directives in the Setup section. Those things that you are finding are   VersionInfoDescription and VersionInfoProductName.
You shoud put the below to your code.
[Setup]
...
VersionInfoDescription="Put here to show the File Description."
VersionInfoProductName="Put here to show the Product Name."
...


text:
University of Nebraska Board of Regents to make campus visit to UNL
The University of Nebraska Board of Regents will make a campus visit to the University of Nebraska-Lincoln on Thursday, Oct. 30, 2014.
The Board will tour several facilities and hear about various education and research efforts happening on campus, among other presentations.
The full itinerary for Thursday's campus visit is as follows:
9:15 a.m.: College of Engineering
Location: Nebraska Hall
Presenter: Tim Wei, dean, UNL College of Engineering
10:15 a.m.: Growing Needs and Implications for the College of Education & Human Sciences
Location: Henzlik Hall/Mabel Lee Hall
Presenters: Marjorie Kostelnik, dean, UNL College of Education & Human Sciences, and John Ingram, associate athletic director for capital planning and construction
11 a.m.: The Future Vision of Textron
Location: Whittier Research Center
Presenter: Bill Nunez, associate to the chancellor
Whittier Research Center Tour
Presenters: Prem Paul, vice chancellor for research and economic development, and Mike Zeleny, assistant vice chancellor for research
Noon: Why Honors? (tour, lunch with honors students, and presentation)
Location: Neihardt Residence Hall
Presenters: Patrice Berger, director, UNL Honors Program, and Sue Gildersleeve, director of housing administration
1:45 p.m.: Brace Hall Tour of Classroom Renovations
Location: Brace Hall
Presenter: Lance Perez, associate vice chancellor and dean of graduate studies
2:30 p.m.: East Campus Enhancements
Location: East Campus Student Union Dining Room
Presenters: Ronnie Green, vice president and vice chancellor for the Institute of Agriculture and Natural Resources, and Sue Gildersleeve, director of housing administration
3 p.m.: Campus visit concludes


text:
Home Breaking News 2 Die in Whittier Collision on Deveron Drive and Beverly Boulevard
2 Die in Whittier Collision on Deveron Drive and Beverly Boulevard
2 Killed in Whittier Crash on Deveron Drive and Beverly Boulevard
Whittier, California – Two people died following a Whittier collision that occurred on Deveron Drive and Beverly Boulevard early Sunday morning.
According to authorities, a car lost control and crashed into two parked vehicles just before 3:30 a.m.
The impact ejected one of the vehicle's occupants, who died at the scene.
A second party inside the vehicle also died following the crash.
Two other people were taken to a hospital with unspecified injuries.
The cause of the crash is under investigation.
Arin Khodaverdian, CEO and Managing Attorney of Alpine Law Group, urges accidents victims and their families to immediately seek legal assistance. "Those who are affected by tragedy have rights, but they must act quickly in order to preserve those rights. Swift action by an intelligent and experienced attorney is imperative in order to preserve evidence, interview witnesses and immediately begin pursuing any and all responsible parties."
"Usually, clients hire me immediately after their first consultation because they quickly figure out that I will treat them like family. I lift a weight off their shoulders immediately. That is my job. We simply leave no stone unturned – period. We go after everyone and everything. There is no other way to ensure justice," Mr. Khodaverdian states.
While Mr. Khodaverdian does not represent any of the parties involved in this incident, he has recovered over $100 million for thousands of injury victims and their families. "In one case, we were able to recover a 7-figure settlement within 11 business days of being hired by the family of a victim killed by a negligent driver. We have numerous other 6-figure settlements within 30 days. While these results are atypical and do not predict future outcomes, justice in these cases helped bring about closure and a swift sense of justice for the families."
Mr. Khodaverdian is the recipient of numerous accolades including one of the: Top 10 Wrongful Death Settlements in CA in 2019; Top 10 Motor Vehicle Accident Settlements in CA in 2019; Top 10 Truck Accident Settlements in CA in 2019, and Top 10 Motorcycle Accident Settlements in CA in 2019. He is a Berkeley graduate and proud personal injury and wrongful death attorney. He is personally available 24/7 for free, no-obligation consultations and second opinions by calling 800-984-4123. He can also be reached via email using the contact form on this site.
2 Killed in Whittier Crash
Two Killed in Whittier Crash
Whittier Collision
Whittier Collision on Deveron Drive and Beverly Boulevard
Whittier Collision Sunday Morning
Previous articleJoshua Ghziouna Killed in Madera County Crash on Road 222 near Bonnie B Road
Next article1 Hospitalized after El Monte Crash on Sunday Morning


text:
James Corden becomes 6th Backstreet Boy in deeply sexual performance
By Lindsay Davis 2016-09-29 10:02:07 UTC
James Corden just wants boy bands to be a thing again.
And by boy band, he doesn't mean those chaps with the too-tight jeans, soulful expressions and ironic facial hair (we're looking at you, 1D).
No, we're talking synchronised dancing, matching outfits, and incomprehensible quantities of hair gel — basically, all the cheese that the '90s and early 2000s have to offer.
SEE ALSO: James Corden, 'Late Late Show' staff take on Usain Bolt in epic race
Fortunately for us, James Corden is a man of action.
The Late Late Show host surprised his audience with a performance from arguably the most iconic boy band of all time: the Backstreet Boys.
They're still active, and in 2017 they will begin a residency at The AXIS Planet Hollywood in Las Vegas.
Tiny chihuahua looks more like a porcupine after a run-in with static electricity
Fierce trailer for Netflix's 'I Care A Lot' debuts Rosamund Pike's cool new villain
NASA drops an unsettling new video
Tom Holland suits up as a tortured soldier in gritty 'Cherry' trailer
Topics: Backstreet Boys, boy band, Comedian, Culture, humor, james corden, las vegas, Music, the late late show, UK


With only two more weeks of racing before the MYC Sunfish season finale Annual Regatta, the Sunfish racing is as tight as ever. This week saw Rick Schlosser, Bernadette Levesque and Mark Stoughton jockeying for leads in every race in extremely shifty and puffy conditions. Winds ranged from 3 to 15 mph, and 30 degree wind shifts were common. Leads changed by the leg, and occasionally by the wind shift. The person who was ahead at the first mark was not necessarily in the lead at the second mark, and places frequently changed within boat lengths of the finish line.
We've got two more weeks of racing, then a break for the Labor Day holiday weekend, and the season finishes up with the 64th MYC Annual Regatta on Sept 6-8.
We've Poached Another Laser Sailor!
Where O Where Have our Updates Gone?


text:
The Lives of a Bengal Lancer (1935)
Watch The Lives of a Bengal Lancer (1935) Online
In the Northwest Frontier of India, the 41st Bengal Lancers leaded by the harsh Colonel Tom Stone are having trouble with the rebellious leader Mohammed Khan. After two casualties, the experienced but insubordinate Lieutenant Alan McGregor receives as replacement, the arrogant Lieutenant Forsythe and the immature son of Colonel Stone, Lieutenant Donald Stone. With the intention to prove that he will not have any privilege in the troop, the reception of Colonel Stone to his son is absolutely cold, but he becomes the protégé of McGregor. When Lieutenant Stone is kidnapped by Mohammed Khan, McGregor and Forsythe disobey the direct order of their commander, disguise as Indian peddlers and go to Khan's fortress to attempt to rescue their friend
Genre: Adventure, Drama, War
Director: Henry Hathaway
Actors: C. Aubrey Smith, Douglass Dumbrille, Franchot Tone, Gary Cooper, Guy Standing, Kathleen Burke, Richard Cromwell
The Ant Bully (2006)
Fed up with being targeted by the neighborhood bully, 10-year-old Lucas Nickle vents his frustrations on the anthill in his front yard … until the insects shrink him to the…
With the help of his loyal protector Butler, 12-year-old genius Artemis Fowl, descendant of a long line of criminal masterminds, seeks to find his mysteriously disappeared father, and in doing…
Genre: Adventure, Family, Fantasy, Science Fiction, Trending
Snow White: A Tale of Terror (1997)
When young Lili's mother dies in childbirth, her father remarries Lady Claudia, a woman ruled by an evil mirror with the power to make her queen of all living things….
Genre: Adventure, Fantasy, Horror
Frank Corvin, 'Hawk' Hawkins, Jerry O'Neill and 'Tank' Sullivan were hotdog members of Project Daedalus, the Air Force's test program for space travel, but their hopes were dashed in 1958…
Outcast of the Islands (1951)
After financial improprieties are discovered at the Eastern trading company where he works, Peter Willems flees the resulting disgrace and criminal charges. He persuades the man who gave him his…
Fantasy Island (2020)
A group of contest winners arrive at an island hotel to live out their dreams, only to find themselves trapped in nightmare scenarios.
Genre: Adventure, Fantasy, Horror, Mystery, Science Fiction
Cannibal Women in the Avocado Jungle of Death (1989)
The U.S. government, eager to protect the nation's avacado supplies, recruits feminist professor Margo Hunt to make contact with the Piranha Women, an all-female tribe who believe men are only…
Genre: Action, Adventure, Comedy, Horror, Romance
Barely Lethal (2015)
A 16-year-old international assassin yearning for a "normal" adolescence fakes her own death and enrolls as a senior in a suburban high school. She quickly learns that being popular can…
Target (1985)
A Texan with a secret past searches Europe with his son after the KGB kidnaps his wife.
Genre: Action, Adventure, Crime, Drama, Mystery, Thriller
Nausicaä of the Valley of the Wind (1984)
After a global war, the seaside kingdom known as the Valley of the Wind remains one of the last strongholds on Earth untouched by a poisonous jungle and the powerful…
Genre: Adventure, Animation, Fantasy, Science Fiction
Hardcore Henry (2015)
Henry, a newly resurrected cyborg who must save his wife/creator from the clutches of a psychotic tyrant with telekinetic powers, AKAN, and his army of mercenaries. Fighting alongside Henry is…
Country: China, Russia, USA
Ice Age: The Meltdown (2006)
Diego, Manny and Sid return in this sequel to the hit animated movie Ice Age. This time around, the deep freeze is over, and the ice-covered earth is starting to…
Trailer: The Lives of a Bengal Lancer (1935)