text:
\section{TODO}

\section{Introduction}

Gaussian processes (GPs) are flexible, non-parametric models often used in regression and classification tasks \citep{rasmussen2006gaussian}. They are probabilistic models and provide both a prediction and an uncertainty quantification. For this reason, GPs are a common choice in different applications see, e.g. \citet{Shahriari.etal2016}, \citet{Santner2018} and \citet{Hennig.eal2015}.  The flexibility of GPs, however, comes with an important computational drawback: training requires the inversion of a $N \times N$ matrix, where $N$ is the size of the training data, resulting in a $O(N^3)$ computational complexity. Many approximations that mitigate this issue have been proposed, see,~\citet{Liu.etal2018} for a review.

Sparse inducing points approaches \citep{quinonero2005unifying} have long been employed for large data \citep{csato2002sparse,seeger2003fast,snelson2006sparse}. The core idea of such methods is to approximate the unknown function with its values at few, $M \ll N$, well-selected input locations called inducing points which leads to a reduced computational complexity of $O(NM^2)$. In \citet{titsias2009variational}  a variational method was introduced that keeps the original GP prior and  approximates the posterior with variational inference. This method guarantees that by increasing the number of inducing inputs the approximate posterior distribution is closer to the full GP posterior in a Kullback-Leibler divergence sense.

An alternative method for variational inference on sparse GPs was introduced in \citet{ChengBoots2016}, where the authors proposed a variational inference method based on a property of the reproducing kernel Hilbert space (RKHS) associated with the GP. The main idea is to write the variational problem in the RKHS associated with the GP and to parametrize the variational mean and covariance accordingly. This method has been improved in several works \citep{ChengBoots2017,Salimbeni.etal2018} that provide more powerful formulations for the mean and covariance of the variational distribution. In particular, \citet{Salimbeni.etal2018} proposed a powerful orthogonally decoupled basis that allowed for an efficient natural gradient update rule.

A parallel line of research for sparse GPs studies inter-domain approximations. Such approaches exploit spectral decompositions of the GP kernel and provides low rank approximations of the GP, see \citet{RahimiRecht2007,LazaroGredilla.etal2009,SolinSarkka2014,Hensman.etal2018}. Inter-domain methods replace inducing points variables with more informative inducing features which, usually, do not need to be optimized at training time, thus potentially reducing the computational cost of the approximation method. \citet{Hensman.etal2018} combined the power of an inter-domain approach with the variational setup of \citet{titsias2009variational} in their variational Fourier feature method.

In this work we combine the flexibility of the orthogonally decoupled RKHS bases introduced in \citet{Salimbeni.etal2018} and the explanatory power of inter-domain approaches to propose a new method for training sparse Gaussian processes. We build a variational distribution parametrized in the mean by an inducing point basis and in the covariance by a variational Fourier features basis. Since variational inference for the basis parameterizing the mean does not require a matrix inversion we can use a large number of inducing points and obtain an approximation close to the true posterior mean. On the other hand, by using a variational Fourier features basis in the covariance, we exploit the higher informative power of such features to obtain better covariance estimates. The orthogonal structure guarantees that the range of the two bases does not overlap.

Sect.~\ref{sec:IntroGP} reviews Gaussian process for regression and classification and recalls the RKHS property exploited by our approximation. In Sect.~\ref{sec:sparseRKHS} we review the previously proposed techniques for sparse GPs with RKHS bases. We propose our novel technique and we describe the implementation details in Sect.~\ref{sec:ODVFF}. We test our method on synthetic and real data in Sect.~\ref{sec:experiments} and we discuss advantages and drawbacks in Sect.~\ref{sec:discussion}.

\section{Gaussian processes}
\label{sec:IntroGP}

A real valued Gaussian process $f(x) \sim GP(m(x),k(x,x))$, defined on an input space $\mathcal{X} \subset \mathbb{R}^D$, is a stochastic process such that for any $N>0$ the values $\{f(x_i): x_i\in \mathcal{X} \}_{i=1,\ldots, N}$ follow a multivariate normal distribution. It is completely characterized by its mean function $m(x) := \mathbb{E}[f(x)]$ and its covariance kernel $k(x,x^\prime) := \operatorname{Cov}(f(x),f(x^\prime))$. The covariance kernel $k$ is a positive-definite function and a reproducing kernel in an appropriate Hilbert space, see, e.g.,~\citet{BerlinetThomasAgnan2004}. The reproducing property of $k$ implies that there exists a Hilbert space $\mathcal{H}$ such that
\begin{equation*}
k(x,x^\prime) = <\psi(x), \Sigma \psi(x^\prime)>_{\mathcal{H}}, \qquad x,x^\prime \in \mathcal{X},
\end{equation*}
where $\psi: \mathcal{X} \rightarrow \mathcal{H}$ is a feature map and $\Sigma: \mathcal{H} \rightarrow \mathcal{H}$ is a bounded positive semi-definite self-adjoint operator. Moreover if $m \in \mathcal{H}$, then we can associate a function $\mu \in \mathcal{H}$ such that $m(x) = <\psi(x),\mu>_{\mathcal{H}}$. The couple $\mu, \Sigma$ is a dual representation of a Gaussian process with mean $m$ and covariance $k$ into the space $\mathcal{H}$. Here we follow \citet{Salimbeni.etal2018} and, for simplicity, we denote by $f \sim GP_\mathcal{H}(\mu,\Sigma)$ the dual representation of the $GP(m,k)$ in the Hilbert space $\mathcal{H}$. This notation is only used here to denote that the objects $\mu,\Sigma$ have the dual role of $m, k$, however it does not mean that the GP samples belong to $\mathcal{H}$.

Let us denote by $\mathbf{y} = (y_1, \ldots, y_N)$, a vector of $N$ output values and by $\mathbf{X} = (x_1, \ldots, x_N )^T$ a matrix of inputs. We consider a likelihood model, not necessarily Gaussian, that factorizes over the outputs, i.e. $p(\mathbf{y} \mid \mathbf{f}) = \prod_{i=1}^N p(y_i \mid f(x_i))$, where $\mathbf{f} = (f(x_i))_{i=1}^N$ and $f\sim GP(m(x),k(x,x^\prime) )$ for a prior mean function $m$ and covariance kernel $k$. The covariance kernel $k$ is often chosen from a parametric family such as the squared exponential or the Mat\'ern family, see \citet{rasmussen2006gaussian}, chapter~4.
The GP provides a prior distribution for the latent values, i.e. $\mathbf{f} \sim N(0,K_N)$ and we can use Bayes rule to compute the posterior distribution $p(\mathbf{f} \mid \mathbf{y}) = \frac{p(\mathbf{y} \mid \mathbf{f}) p(\mathbf{f})}{p(\mathbf{y})}$.

Consider now a regression example where we have a training set $\mathcal{D} = \left\{ y_i, x_i \right\}_{i=1}^N=\left(\mathbf{y},\mathbf{X}\right)$ of $N$ pairs of inputs $x_i\in \mathbb{R}^D$ and noisy scalar outputs $y_i$ generated by adding independent Gaussian noise to a latent function $f(x)$, that is $y_i = f(x_i)+\varepsilon_i$, where $\varepsilon_i\sim \mathcal{N}(0,\sigma_n^2)$. The posterior distribution of the Gaussian process $p(\mathbf{f} \mid  \mathbf{y})$ is Gaussian with mean and covariance given by
\begin{align*}
m_N(x) &= m(x) + k(x,\mathbf{X}) [k(\mathbf{X})+\sigma_{n}^2 I]^{-1}(\mathbf{y} - m(\mathbf{X})) \\
k_N(x,x^\prime) &= k(x,x^\prime) - k(x,\mathbf{X}) [k(\mathbf{X})+\sigma_{n}^2 I]^{-1} k(\mathbf{X},x^\prime)
\end{align*}
where $k(\mathbf{X}) = [k(x_i,x_j)]_{i,j =1, \ldots, N} \in \mathbb{R}^{N \times N}$, $k(x,\mathbf{X}) = [k(x,x_i)]_{i=1, \ldots, N} \in \mathbb{R}^{1\times N}$, $k(\mathbf{X},x)=k(x,\mathbf{X})^T$ and $m(\mathbf{X}) = [m(x_i)]_{i=1,\ldots,N}$.

As mentioned above there exists a Hilbert space $\mathcal{H}$ with inner product $<\cdot, \cdot>_\mathcal{H}$ and a feature map $\psi(x)$ such that $k(x,x^\prime) = <\psi(x), \Sigma\psi(x^\prime)>_\mathcal{H}$. In this case, the operator is $\Sigma = I$ and we further assume that $m(x) = <\psi(x),\mu>_\mathcal{H}$ for some $\mu \in \mathcal{H}$. As an example, we can choose the canonical feature map $\psi(x) = k(x,\cdot)$ and $\mathcal{H} = \mathcal{H}_k$, the RKHS associated with $k$. The posterior mean and covariance above can be rewritten as $m_N(x) = <\psi(x),\mu_N>_{\mathcal{H}}$ and $k_N(x,x^\prime) = <\psi(x),\Sigma_N\psi(x^\prime)>_{\mathcal{H}}$ where
\begin{align}
\label{eq:muSigma}
\mu_N &:= \mu + \Psi_X [k(\mathbf{X})+\sigma_{n}^2 I]^{-1} (\mathbf{y} - m(\mathbf{X})) \\
\Sigma_N &:= I-\Psi_X [k(\mathbf{X})+\sigma_{n}^2 I]^{-1} \Psi_X^T.
\end{align}
where $\Psi_X = [\psi(x_1), \ldots, \psi(x_N)]$ and we use the notation
\begin{align*}
\Psi_X^T \psi(x) :&= [<\psi(x_1),\psi(x)>_\mathcal{H}, \ldots, <\psi(x_N),\psi(x)>_\mathcal{H}]^T.
\end{align*}

Note that, in this example, we have $\Psi_X^T \psi(x)= k(\mathbf{X},x)$.
In the dual representation then the prior GP corresponds to $f \sim GP_\mathcal{H}(\mu,I)$ and the posterior GP to $f \sim GP_\mathcal{H}(\mu_N,\Sigma_N)$ with $\mu_N,\Sigma_N$ as in eq.~\eqref{eq:muSigma}.

From the equations above we can  see that GP training involves the inversion of a matrix of size $N\times N$, thus requiring $O(N^3)$ time. In what follows, we always assume $m(x) \equiv 0$.

\section{Sparse GP and RKHS basis}
\label{sec:sparseRKHS}

In order to reduce the computational cost, we follow here the variational approach introduced in \citet{titsias2009variational} and then generalized to stochastic processes in \citet{Matthews.etal16}. The idea is to approximate the posterior distribution $p(f \mid \mathbf{y})$ with a variational distribution $q(f)$. Since the posterior is a GP, the variational distribution should also be a GP. The optimal distribution is then selected as
\begin{align*}
q &= \arg \min_q KL( q(f) \Vert p(f\mid \mathbf{y})) \\
&= \arg\min_q \mathbb{E}_q[\log q(f) - \log p(f \mid \mathbf{y})].
\end{align*}
In \citet{titsias2009variational,Matthews.etal16}, the authors follow the sparse inducing points approach \citep{quinonero2005unifying} and consider $q(f,\mathbf{u}) = q(f \mid \mathbf{u}) q(\mathbf{u})$ parametrized by $\mathbf{u} = [f(r_i)]_{i=1, \ldots,M}$ a vector of $M \ll N$ inducing outputs evaluated at inputs $r_1, \ldots, r_M \in \mathcal{X}$. The resulting conditional distribution is
\begin{align*}
q(f(x) \mid \mathbf{u}) = GP\bigg(k(x,\mathbf{u}) K_{\mathbf{u},\mathbf{u}}^{-1} \mathbf{u},\quad k(x,x) - k(x,\mathbf{u}) K_{\mathbf{u},\mathbf{u}}^{-1}k(\mathbf{u},x) \bigg).
\end{align*}

The joint distribution $q(f,\mathbf{u})$ is optimized, by selecting the variational parameters $b \in \mathbb{R}^M$ and $S \in \mathbb{R}^{M \times M}$ of the variational distribution $q(\mathbf{u}) = N(b,S)$. In the regression case \citep{titsias2009variational}, the optimal $q(\mathbf{u})$ has analytical expressions for its mean and covariance. Stochastic optimization techniques and mini-batch training were developed for regression \citep{hensman2013gaussian,Schuerch.etal2019} and classification \citep{Hensman.eal15}.

\subsection{Variational problem in the RKHS space}
In the dual view presented in Sect.~\ref{sec:IntroGP} the posterior distribution can be represented as $f\mid \mathbf{y} \sim GP_\mathcal{H}(\mu_N,\Sigma_N)$. The variational problem can also be represented in this dual form. In particular here the distribution $q$ is represented as $q_{\mathcal{H}}(f) = GP_\mathcal{H}(\mu, \Sigma)$ and we would like to find the distribution $q_{\mathcal{H}}(f)$ that minimizes
\begin{align}
\mathcal{L}(q_{\mathcal{H}}) = - \sum_{i=1}^{N} \mathbb{E}_{q_{\mathcal{H}}(f(x_i))}[ \log p(y_i \mid f(x_i)) ] + KL(q_{\mathcal{H}}(f) \Vert p(f))
\label{eq:lowBound}
\end{align}
where $p(f) \sim GP_{\mathcal{H}}(0,I)$. It can be shown \citep{ChengBoots2016,ChengBoots2017,Salimbeni.etal2018} that $\mathcal{L}(q_{\mathcal{H}}) = KL(q_{\mathcal{H}}(f) \Vert p_{\mathcal{H}}(f \mid \mathbf{y}) )$ up to a constant. In this dual formulation, the objects $\mu,\Sigma$ are a function and an operator over an Hilbert space respectively and they cannot be optimized directly. In order to optimize $\mu$ and $\Sigma$ we need to choose an appropriate parametrization. \citet{ChengBoots2016}, first proposed the following decoupled decomposition, inspired by eq.~\eqref{eq:muSigma},
\begin{equation}
\mu = \Psi_\alpha a, \qquad \Sigma = I - \Psi_\beta A \Psi_\beta^T,
\label{eq:fullydecNoInv}
\end{equation}
where $\alpha, \beta$ are sets of inducing variables, $a \in \mathbb{R}^{\lvert \alpha \rvert}$, $A \in \mathbb{R}^{\lvert \beta \rvert \times \lvert \beta \rvert}$ are variational parameters and $\Psi_\alpha$ is a basis functions vector defined as $\Psi_\mathbf{u}= [k(r_1, \cdot), \ldots, k(r_M,\cdot)]^T$, where $u_i =f(r_i) = \alpha_i$ for $\Psi_\alpha$ or $u_i = \beta_i$ for $\Psi_\beta$. If we choose $\alpha = \beta=\mathbf{u}$, $a = K_{\mathbf{u},\mathbf{u}} b$ and $A= -K_{\mathbf{u},\mathbf{u}}^{-1} (S - K_{\mathbf{u},\mathbf{u}} )K_{\mathbf{u},\mathbf{u}} ^{-1}$ we obtain the standard  \citep{titsias2009variational} result where $b,S$ denote the usual variational parameters to be optimized.

\subsection{Orthogonally decoupled bases}
\label{subsec:ODB}

The decoupled parametrization~\eqref{eq:fullydecNoInv} was shown to be insufficiently constrained in \citet{ChengBoots2017}. If the size of $\beta$ is increased, the basis in \eqref{eq:fullydecNoInv} is not necessarily more expressive and, in particular, the mean does not necessarily improve.

In~\citet{ChengBoots2017}, the authors further generalized~\eqref{eq:fullydecNoInv} with a hybrid basis that addressed this issue, however the optimization procedure for this basis was shown to be ill-conditioned. Finally, \citet{Salimbeni.etal2018} proposed the following orthogonal bases decomposition.
\begin{align}
\label{eq:orthDecMean}
\mu &= (I - \Psi_\beta K_\beta^{-1}\Psi_\beta^T)\Psi_\gamma a_\gamma +\Psi_\beta a_\beta \\
\Sigma &= I - \Psi_\beta K_\beta^{-1} \Psi_\beta^T + \Psi_\beta K_\beta^{-1} S K_\beta^{-1}\Psi_\beta^T .
\label{eq:orthDecCov}
\end{align}
Pre-multiplying $\Psi_\gamma$ by $(I - \Psi_\beta K_\beta^{-1}\Psi_\beta)$ makes the two bases orthogonal and the optimization problem well-conditioned. In practice the mean function $\mu$ is now decomposed into bases which are no longer overlapping therefore the variational space can be explored more efficiently by the optimizer.

We propose here to replace the basis functions parametrized by the inducing points $\beta$ with the ones build on RKHS inducing features \citep{Hensman.etal2018}.

\subsection{Inter-domain approaches}
\label{subsec:interDomain}

An inducing output $u_i$ can be seen as the result of the evaluation functional $L_{r_i}[f]= f(r_i)$, where $u_i = f(r_i)$ and $r_i$ is the inducing input corresponding to the inducing output $u_i$. As long as the resulting random variable $L_{r_i}[f]$ is well defined, we can extend this approach to more general linear functionals; inter-domain sparse GPs are built on this general notion. For example, approaches based on Fourier features \citep{RahimiRecht2007,LazaroGredilla.etal2009} choose the functional $L_\omega[f] = \int f(x) e^{-i\omega x} dx$.

Here we start by considering only one-dimensional inputs and, by following \citet{Hensman.etal2018}, we restrict the input domain to an interval $[\kappa_0,\kappa_1]\subset \mathbb{R}$.
While this might seem like a strong restriction, in practice data is always observed in a finite window and we can select a larger interval $[\kappa_0,\kappa_1]$ that includes all training inputs and the extrapolation region of interest. Moreover, instead of considering an $L^2(\kappa_0,\kappa_1)$ inner product with Fourier features, we consider the RKHS inner product $<\cdot, \cdot>_{\mathcal{H}_k}$, i.e. we consider the RKHS features \citep{Hensman.etal2018}, defined as

\begin{equation}
\zeta_i := <f, \phi_i>_{\mathcal{H}_k}
\label{eq:VFfeature}
\end{equation}
where $<\cdot, \cdot>_{\mathcal{H}_k}$ is the inner product in $\mathcal{H}_k$, the RKHS induced by the GP covariance kernel $k$, and
\begin{align}
\nonumber
\Phi = [&\phi_i]_{i=1}^{2M+1} \\
= [&1, \cos(\omega_1(x-\kappa_0)), \ldots, \cos(\omega_M(x-\kappa_0)), \\
&\sin(\omega_1(x-\kappa_0)), \ldots \sin(\omega_M(x-\kappa_0))].
\label{eq:trunFourier}
\end{align}

The covariance between inducing variables and function values and the cross-covariances between inducing variables can be written \citep{Hensman.etal2018} as
\begin{align}
\label{eq:rkhsFeatProperty1}
\operatorname{Cov}(\zeta_i,f(x)) &= \phi_i(x) , \\
\operatorname{Cov}(\zeta_i,\zeta_{i^\prime}) &= <\phi_i,\phi_{i^\prime}>_{\mathcal{H}_k}.
\label{eq:rkhsFeatProperty2}
\end{align}
These properties allow for a fast computation of the kernel matrices as long as the matrix $K_{\phi,\phi}= [<\phi_i,\phi_{i^\prime}>_{\mathcal{H}_k}]_{i,i^\prime=1,\ldots,2M+1}$ is finite and can be computed. Analytical formulae are available \citep{Hensman.etal2018} to compute the inner product above if $k$ is a Mat\'ern kernel with smoothness parameter $\nu=1/2,3/2, 5/2$ defined on an interval $[a, b] \subset \mathbb{R}$.
The method requires the space $\mathcal{F} = span(\Phi)$ to be a subspace of the RKHS $\mathcal{H}_k$. This property is not always true, notable counterexamples \citep{Hensman.etal2018} are the Mat\'ern RKHS on $\mathbb{R}$ and the RBF and Brownian motion kernel on $[0,1]$.

In the cases where analytical formulae are available, the Gram matrix $K_{\phi,\phi}$ also has the striking property that it can be written as a diagonal matrix plus several rank-one matrices. This allows for more efficient computation of matrix product such as $K_{\phi,\phi}^{-1} K_{\phi,f}$, see \citet{Hensman.etal2018}.

\begin{figure}[ht]

	\includegraphics[width=\linewidth]{Cbeta1d_ID40_rep2_3Methods.png

	\caption{Comparison of different methods, no training data in $[0.45,0.55]$, $\abs[\beta]+\abs[\gamma]=200$.}
	\label{fig:1dExampleComparison}
\end{figure}

\section{Orthogonally decoupled variational Fourier features}
\label{sec:ODVFF}

The orthogonal decomposition in eq.~\eqref{eq:orthDecMean},~\eqref{eq:orthDecCov} only requires an appropriately defined kernel matrix $K_\beta$ and the operators $\Psi_\gamma, \Psi_\beta$; however they do not have to be generated from inducing points. In practice, \citet{Salimbeni.etal2018} shows that the restriction to disjoint sets of inducing points $\gamma$, $\beta$ helps in the optimization procedure, however, the orthogonally decoupled formulation also enforces that the bases related to $\gamma$ and $\beta$ are orthogonal. We can thus exploit this property to obtain a combination of inter-domain basis and inducing points basis which are mutually orthogonal.

The Orthogonally Decoupled Variational Fourier Features (ODVFF) method considers two sets of variational parameters: a set of inducing points $\gamma = [\gamma_i]_{i=1, \ldots, \abs[\gamma]}$,  and a set of RKHS features $\beta = [ \beta_j]_{j=1, \ldots, \abs[\beta]}$ defined as $\beta_j := <f, \phi_j>_{\mathcal{H}_k}$, with $\phi_j$ defined as in \eqref{eq:trunFourier} for $j=1, \ldots, \abs[\beta]$. We can then build a variational distribution $q$ in the dual space parametrized with $\mu$ and $\Sigma$ defined as

\begin{align}
\label{eq:orthDecVFFmean}
\mu &= (I - \Psi_\beta K_\beta^{-1}\Psi_\beta^T)\Psi_\gamma a_\gamma +\Psi_\beta a_\beta \\
\Sigma &= I - \Psi_\beta K_\beta^{-1} \Psi_\beta^T + \Psi_\beta K_\beta^{-1} S K_\beta^{-1}\Psi_\beta^T .
\label{eq:orthDecVFFvar}
\end{align}
with
\begin{align*}
\Psi_\beta &= [\operatorname{Cov}(\beta_1,f(\cdot)), \ldots, \operatorname{Cov}(\beta_{\lvert \beta \rvert},f(\cdot))]^T  \\
&= [ \phi_1(\cdot), \ldots, \phi_{\lvert \beta \rvert}(\cdot)]^T = \Phi,
\end{align*}
and $K_\beta = K_{\phi,\phi} = [\operatorname{Cov}(\beta_i,\beta_{i^\prime})]_{i,i^\prime=1,\ldots, \abs[\beta]}$ is the Gram matrix obtained from the cross-covariance between inducing features. Note that both $\Psi_\beta$ and $K_\beta$ are easily computable by exploiting~\eqref{eq:rkhsFeatProperty1} and~~\eqref{eq:rkhsFeatProperty2}.  $\Psi_\gamma$ is instead obtained from the inducing points basis $[\gamma_i]$ as $\Psi_\gamma(\cdot) = [k(\gamma_1,\cdot), \ldots, k(\gamma_{\abs[\gamma]},\cdot) ]$.

The distribution above depends on the variational parameters $a_\gamma, a_\beta$ and $S$ which can be either optimized analytically or with stochastic gradient descent and natural gradients, see~\citet{Salimbeni.etal2018b}.  Moreover~\eqref{eq:orthDecVFFmean} and \eqref{eq:orthDecVFFvar} require choosing the hyper-parameters that build the basis $\gamma$ and $\beta$. Note that the parametrization defines $\Sigma$ with hyper-parameters that do not need optimization, in fact the inducing Fourier features $\beta$ are fixed and chosen in advance. This reduces the size of the optimization problem compared to an orthogonally decoupled model with two sets of inducing points.

We consider here $\abs[\beta] = 2F+1$, $\Phi$ as in \eqref{eq:trunFourier} and the frequencies $\omega_1, \ldots, \omega_F$, are chosen as harmonic on the interval $[\kappa_0,\kappa_1]$, i.e.  $\omega_i  = \frac{2\pi i}{\kappa_1-\kappa_0}$ $i=1, \ldots, F$. Compared to inducing points orthogonally decoupled basis with the same $\abs[\gamma]$ and $\abs[\beta]$, then we only need to optimize $\abs[\gamma]$ inducing parameters. Moreover for the same number of features $F$, inducing RKHS features have been empirically shown to give better fits than inducing points, see \citet{Hensman.etal2018}. Since we parametrize the covariance of the variational distribution with RKHS features, our ODVFF generally obtains better coverage than equivalent orthogonally decoupled inducing points.

\subsection{Multi-dimensional input spaces}

The variational Fourier features used in the previous section were limited to one dimensional input spaces. Here we exploit the extensions introduced in \citet{Hensman.etal2018} to generalize the method to input spaces of any dimension. In particular we look at additive and separable kernels in an hyper-rectangle input space $\mathcal{X} = \prod_{d=1}^{D}[a_d, b_d]$, $a_d,b_d \in \mathbb{R}$.

\subsubsection{Additive kernels}

The first extension to multiple dimensions can be achieved by assuming that the Gaussian process can be decomposed in an additive combination of functions defined on each input dimension. We can write $f(\mathbf{x}) = \sum_{d=1}^D f_d(x_d)$ where $\mathbf{x} \in \mathcal{X}$, and $f_d \sim GP(0, k_d(x_d,x_d^\prime))$. This results in an overall process defined as
\begin{equation*}
f \sim GP\left(0, \sum_{d=1}^{D} k_d(x_d,x_d^\prime)\right).
\end{equation*}
The function is decomposed on simple one dimensional functions, therefore we can select a Mat\'ern kernel for each dimension and define $D\abs[\beta]$ features
\begin{equation*}
\zeta_{i,d} = <\phi_i, f_d>_{\mathcal{H}_d} \quad i=1, \ldots, \abs[\beta], \quad d=1, \ldots, D,
\end{equation*}
where $\mathcal{H}_d$ is the RKHS associated with the kernel of the $d$th dimension. The Gram matrix $K_{\phi,\phi}$ is then the $D\abs[\beta] \times D\abs[\beta]$ block-diagonal matrix where each block of dimension $\abs[\beta] \times \abs[\beta]$ is the one-dimensional Gram matrix.

\subsubsection{Separable kernels}

An alternative approach involves separable kernels, where the process is defined a
\begin{equation*}
f \sim GP\left(0, \prod_{d=1}^{D} k_d(x_d,x_d^\prime)\right).
\end{equation*}

In this case we parametrize the basis as the Kronecker product of features over the dimensions, i.e.
\begin{equation*}
\Phi = \bigotimes_{d=1}^D [\phi_{d,1}(x_d), \ldots, \phi_{d,2F+1}(x_d)]^T
\end{equation*}
where  $\phi_{d,1}$ is the first feature for the $d$th dimension. This structure implies that each feature is equal to $\prod_{d=1}^{D} \phi_{d,i}$ and that the inducing variables can be written as $\zeta_i = <~\prod_{d=1}^{D} \phi_{d,i}, f~>_\mathcal{H}$, $i=1, \ldots, \abs[\beta]$.

The separable structure results in features that are independent across dimensions, i.e. $\operatorname{Cov}(\phi_{d,i},\phi_{d^\prime, j}) =0$  for all $d \neq d^\prime$ and $i,j =1, \ldots, \abs[\beta]$. Moreover, the Gram matrix corresponding to each dimension is the same as the one dimensional case. The overall Gram matrix $K_{\phi,\phi}$ is then a block-diagonal matrix  of size $\abs[\beta]^D \times \abs[\beta]^D$ where each diagonal block of size $\abs[\beta] \times \abs[\beta]$ is the Gram matrix corresponding to a one dimensional problem. As in the one dimensional case the covariance between function values and inducing variables is $\operatorname{Cov}(\zeta_i, f(x)) = \phi_i(x)$.

Separable kernels scale exponentially in the input dimension $D$. For this reason they are not practically usable for dimensions higher than $3$. Additive kernels on the other hand do not suffer from this problem, however they have a reduced explanatory power because of the strong assumption of independence across dimensions.

\subsection{Variational parameters training}
In order to train the ODVFF model we need to find the variational parameters $a_\gamma, a_\beta, S$ in \eqref{eq:orthDecMean},~\eqref{eq:orthDecCov} such that the quantity in eq.~\eqref{eq:lowBound}, the negative ELBO, is minimized. If the likelihood is Gaussian, the parameters have an analytical expression
Such expressions however require matrix inversions of sizes $\abs[\gamma] \times\abs[\gamma]$ and $\abs[\beta] \times \abs[\beta]$ as shown in
 \citet{Salimbeni.etal2018}.

On the other hand, stochastic mini-batch training requires only the inversion of $K_{\beta}$ at the additional cost of having to numerically optimize the parameters. Since often
the additional matrix inversion becomes costly, here we only train our models by learning the variational parameters.
Moreover, here we also exploit the natural parameter formulation introduced in \citet{Salimbeni.etal2018,Salimbeni.etal2018b} for faster and more stable training.

\begin{figure}
	\hspace{0.1in}
	\includegraphics[width=0.8\linewidth]{./Betas1d_ID40_logLik.png}
	\caption{Mean test log-likelihood, $15$ replications. $\abs[\beta]+\abs[\gamma]=200$.}
	\label{fig:1dEx_avgLogLik}
\end{figure}

\subsection{Choice of number of features}
The ODVFF method requires a choice of the number of Fourier features $\abs[\beta]$ and inducing points features $\abs[\gamma]$ to use. This choice involves a trade-off between the cost of inverting the matrix $K_{\beta}$ and that of solving the optimization problem for $\abs[\gamma]$ inducing points.

Figure~\ref{fig:1dExampleComparison} shows a comparison of ODVFF against Variational Fourier Features (VFF) \citep{Hensman.etal2018} and a full-batch variational approach (SGPR) \citep{titsias2009variational}. In this example the data was generated with a GP with mean zero and covariance Mat\'ern with smoothing parameter $\nu=3/2$, length scale $\ell=0.1$ and noise variance $\sigma_n^2 =0.15$. By increasing $\abs[\beta]$ from $21$ (on the left) to $181$ (on the right) the root mean square error (RMSE) of the prediction decreased by $29\%$ and the mean coverage increased from $92\%$ to $96\%$. The confidence intervals at $95\%$ obtained with ODVFF and $\abs[\beta]=181$ become almost indistinguishable from the confidence intervals obtained with SGPR. Note, that SGPR here is trained with the full batch while ODVFF is trained with mini-batches and scales to much larger datasets.

Figure~\ref{fig:1dEx_avgLogLik} shows the average test log-likelihood of ODVFF as a function of $\abs[\beta]$ over $15$ replications of the experiment introduced above. For reference we also plot the average test log-likelihood obtained with VFF and with SGPR.

The improvement in test log-likelihood when $\abs[\beta]$ is increased is mainly driven by a better covariance approximation as indicated by the mean coverage (at $95\%$) on test data which is $98\%$ when $\abs[\beta]=21$ and $96\%$ when $\abs[\beta]=181$.

Table~\ref{tab:synth} shows a comparison between ODVFF and orthogonally decoupled sparse GP (ODVGP) \citep{Salimbeni.etal2018} on a synthetic datasets generated as described in Sect.~\ref{subsec:synthetic}. We train models with $N=50,000$, $\abs[\gamma]+\abs[\beta]=200$ and we consider three choices for $\abs[\beta]$: 11, 99, 189. For each choice we replicate the experiment $10$ times. In this and all following tables bold font highlights the best value. Note that for all dimensions we obtain a better model with a smaller $\abs[\beta]$. Moreover, ODVFF performs better than ODVGP with small $\abs[\beta]$, indicating that the Fourier feature parametrization of the covariance is more powerful. This advantage is reduced when $\beta$ increases as most of the fitting is done with $\gamma$ and the mean parametrization is the same between the two methods.

{\tiny
	\begin{table}
		\caption{Synthetic data set. Mean test log-likelihood values (rank) over $10$ repetitions, higher values denote a better fit.
		\label{tab:synth}
		\begin{center}
			\scriptsize
			\begin{tabular}{lrrrr}
				\toprule
				$D$ & & $\abs[\beta]=11$ &        $\abs[\beta]=99$ & $\abs[\beta]=189$  \\

				\midrule

				$2$ & \begin{tabular}{@{}c@{}} ODVFF \\ ODVGP\end{tabular}  & \begin{tabular}{@{}c@{}} \textbf{-2.834 \rank{1.4}} \\ -2.875 \rank{4.0}\end{tabular} &  \begin{tabular}{@{}c@{}} -2.896	 \rank{4.9} \\ -2.845 \rank{2.2}\end{tabular} & \begin{tabular}{@{}c@{}}
					-2.901 \rank{5.8} \\
					-2.847 \rank{2.7}
				\end{tabular} \\[0.2cm]

				$4$ & \begin{tabular}{@{}c@{}} ODVFF \\ ODVGP\end{tabular} & \begin{tabular}{@{}c@{}} -1.763 \rank{2.8} \\-1.780 \rank{3.8} \end{tabular} & \begin{tabular}{@{}c@{}} -1.796 \rank{4.8} \\ \textbf{-1.751 \rank{1.1}} \end{tabular} & \begin{tabular}{@{}c@{}} -1.923 \rank{5.9} \\ -2.02 \rank{2.6} \end{tabular} \\[0.2cm]

				$8$ & \begin{tabular}{@{}c@{}} ODVFF \\ ODVGP\end{tabular} & \begin{tabular}{@{}c@{}} \textbf{-2.293 \rank{1.0}} \\ -2.356 \rank{2.0}\end{tabular} & \begin{tabular}{@{}c@{}} -3.630 \rank{3.0} \\ -4.887 \rank{4.5}\end{tabular} & \begin{tabular}{@{}c@{}}
					-4.660 \rank{4.3} \\
					-6.173 \rank{5.7}
				\end{tabular} \\

			\end{tabular}
		\end{center}
	\end{table}
}

\section{Experiments}
\label{sec:experiments}

In this section we compare ODVFF with ODVGP, the stochastic variational inference method (SVGP) in \citet{hensman2013gaussian} and with the full-batch variational approach (SGPR) in \citet{titsias2009variational}. We implement our experiments in GPflow \citep{GPflow2017}. We estimate the kernel hyper-parameters and the inducing points locations with the Adam \citep{KingmaBa2015} implementation in Tensorflow. The variational parameters are estimated with Adam and the natural gradient descent method described in \citet{Salimbeni.etal2018b} for SVGP and ODVGP, ODVFF respectively.

\subsection{Synthetic data}
\label{subsec:synthetic}

We test our method on synthetic data generated from Gaussian process realizations. We consider $D=1, 5, 10$ and two different training setups with $N=50,100 \times 10^3$. In all cases the training data is generated as realizations of a zero mean GP with ARD Mat\'ern kernel with smoothing parameter $\nu=3/2$, length scale $\ell=0.1$ for each dimension and variance $\sigma_o^2=1$. The observations are noisy with independent Gaussian noise with variance $\sigma_n^2 =0.2$. We consider a prior GP with mean zero and additive Mat\'ern covariance kernel with $\nu=3/2$. For all dimensions and for all $N$ we optimize the inducing points locations and the hyper-parameters. We fix the total number of inducing parameters to $\abs[\gamma] +\abs[\beta] = 100$ and we consider two scenarios: $\abs[\beta]=10$ and $\abs[\beta]=50$.

Tables~\ref{tab:synth1D}, \ref{tab:synth5D}, \ref{tab:synth10D} show the test log-likelihood values obtained on $1000$ test data points for $D=1,5,10$ respectively. A comparison of the RMSE and the mean coverage values at $95\%$ are reported in appendix. SGPR should be considered as a benchmark as it is the only full-batch method. The other three methods are trained on mini-batches of size $500$ with $8000$ iterations. SVGP and SGPR do not depend on $\abs[\beta]$ but on the overall number of inducing points, thus their likelihood values are simply repeated in those columns. SGPR could not be run in the case $N=100 \times 10^3$ due to memory limitations. ODVFF shows better performance than all other methods with $N=100\times 10^3$ in dimensions $D=5,10$ while it is below SVGP when $D=1$. The difference in RMSE is not very large, while the test mean coverages are significantly different. This further reinforces the idea that a variational Fourier feature parametrization for the covariance allows for better fits.

{\tiny
\begin{table}
	\caption{Synthetic data sets, $D=1$. Average test log-likelihood values (rank) over $20$ repetitions, higher values denote a better fit.
	\label{tab:synth1D}
	\begin{center}
		\scriptsize
		\begin{tabular}{lrrrr}
			& \multicolumn{2}{c}{$N=50000$} & \multicolumn{2}{c}{$N=100000$} \\
			\cmidrule(lr){2-3}
			\cmidrule(lr){4-5}
			& $\abs[\beta]=10$ &        $\abs[\beta]=50$ &        $\abs[\beta]=10$ &        $\abs[\beta]=50$ \\

			\midrule

			ODVFF &   0.085 \rank{2.0} &  \textbf{0.147 \rank{1.5}}&  0.049 \rank{2.20}&  0.040 \rank{2.1} \\

			ODVGP &  0.084 \rank{2.0} &  0.135 \rank{1.7} &  0.090 \rank{2.15}& -0.022 \rank{2.7}\\

			SVGP  &  \textbf{0.090 \rank{2.0}} & 0.090 \rank{2.8} &  \textbf{0.112 \rank{1.65}} & \textbf{0.112 \rank{1.2}} \\

			SGPR & \multicolumn{2}{c}{0.156}  &   \multicolumn{2}{c}{$-$} \\

		\end{tabular}
	\end{center}
\end{table}

}

{\tiny
\begin{table}
	\caption{Synthetic data sets, $D=5$. Average test log-likelihood values (rank) over $20$ repetitions, higher values denote a better fit.
	\label{tab:synth5D}
	\begin{center}
		\scriptsize
		\begin{tabular}{lrrrr}
			& \multicolumn{2}{c}{$N=50000$} & \multicolumn{2}{c}{$N=100000$} \\
			\cmidrule(lr){2-3}
			\cmidrule(lr){4-5}
			& $\abs[\beta]=10$ &        $\abs[\beta]=50$ &        $\abs[\beta]=10$ &        $\abs[\beta]=50$ \\
			\midrule

			ODVFF & \textbf{-2.421 \rank{1.0}} & -4.450 \rank{2.00} & \textbf{-6.278 \rank{1.35}} & \textbf{-4.611 \rank{1.10}} \\

			ODVGP & -2.518 \rank{2.0} & -4.653 \rank{2.25} & -7.885 \rank{2.90} & -5.767 \rank{2.15} \\

			SVGP  & -2.927 \rank{3.0} & \textbf{-2.927 \rank{1.75}} & -6.396 \rank{1.75} & -6.396 \rank{2.75} \\

			SGPR  & \multicolumn{2}{c}{-1.694} &    \multicolumn{2}{c}{$-$} \\

		\end{tabular}
	\end{center}
\end{table}

}

{\tiny
\begin{table}
	\caption{Synthetic data sets, $D=10$. Average test log-likelihood values (rank) over $20$ repetitions, higher values denote a better fit.}
	\label{tab:synth10D}
	\begin{center}
		\scriptsize
		\begin{tabular}{lrrrr}

			& \multicolumn{2}{c}{$N=50000$} & \multicolumn{2}{c}{$N=100000$} \\
			\cmidrule(lr){2-3}
			\cmidrule(lr){4-5}
			& $\abs[\beta]=10$ &        $\abs[\beta]=50$ &        $\abs[\beta]=10$ &        $\abs[\beta]=50$ \\

			\midrule

			ODVFF & \textbf{-2.989 \rank{1.0}} & \textbf{-2.866 \rank{1.1}} & \textbf{-2.640 \rank{1.0}} & \textbf{-2.356 \rank{1.0}} \\

			ODVGP & -3.164 \rank{2.0} & -7.253 \rank{2.9} & -3.233 \rank{2.0} & -9.451 \rank{3.0} \\

			SVGP  & -3.998 \rank{3.0} & -3.998 \rank{2.0} & -4.076 \rank{3.0} & -4.076 \rank{2.0} \\

			SGPR  & \multicolumn{2}{c}{-1.842} & \multicolumn{2}{c}{$-$} \\

		\end{tabular}
	\end{center}
\end{table}
}

\subsection{Benchmarks}

In this section we benchmark the methods on $7$ regression datasets from the UCI Machine Learning repository\footnote{https://archive.ics.uci.edu/ml/index.php} and $10$ datasets from the Penn Machine Learning Benchmark (PMLB) \citep{Olson2017PMLB}. We consider a prior GP with mean zero and additive kernel from the Mat\'ern family with $\nu=3/2$. For all datasets we fix $\abs[\gamma]=300$, $\abs[\beta]=100$, except for kegg\_directed ($\abs[\gamma]=150$, $\abs[\beta]=50$) and sgemmProd ($\abs[\gamma]=120$, $\abs[\beta]=80$).

Table~\ref{tab:uciLogLik} shows the test log-likelihood values obtained on test data selected randomly as $10\%$ of the original dataset with UCI data on the top part and PMLB data in the bottom. Each experiment is repeated $5$ times with different train/test splits. The hyper-parameters, including the inducing points locations, are estimated by maximizing the likelihood of the model for all experiments except for the airline dataset where the inducing points locations are chosen with k-means from the training data. For each model we run the stochastic optimizer for $10000$ iterations with mini-batches of size $400$. We use the same learning rate for all hyper-parameters and a different learning rate in the natural gradient descent for the variational parameters.

{\scriptsize
\begin{table*}
	\caption{Average test log-likelihood (test RMSE) on UCI/PMLB datasets. Higher (lower) values denote a better fit.}
	\label{tab:uciLogLik}
	\begin{center}
		\footnotesize
\begin{tabular}{lrrr}
	\toprule[\heavyrulewidth]

	  & ODVFF &  ODVGP &   SVGP \\

	  \midrul

	  {\scriptsize 3droad (D=3, N=434,874)}  &  \textbf{-1.174 (0.78)} & -1.558 (0.79) & -1.224 (0.79) \\
	  {\scriptsize airline (D=8, N=1,052,631)} & \textbf{-1.881 (0.91)} & -2.546 (0.92)  &	-2.789 \textbf{(0.91)} \\
	  {\scriptsize bike (D=12, N=17,379)}  &  -0.859 \textbf{(0.56)} & \textbf{-0.848 (0.56)} & -0.950 (0.57) \\
	  {\scriptsize kegg\_directed (D=19, N=53,413)} &  0.169 \textbf{(0.09)} & -0.074 (0.11) &  \textbf{0.757} (0.10) \\

	  {\scriptsize protein (D=9, N=45,730)} &  \textbf{-1.221} (0.82) & \textbf{-1.221 (0.81)} & -1.348 \textbf{(0.81)} \\
	  {\scriptsize sgemmProd (D=14, N=241,600)} & -1.180 \textbf{(0.63)} &	\textbf{-1.047} (0.64) &	-7.175 \textbf{(0.63)} \\
	  {\scriptsize tamilelectric (D=2, N=45,781)} &  \textbf{-1.449 (0.99)} & -1.450 \textbf{(0.99)} & -1.492 \textbf{(0.99)} \\
  	  \midrule[\heavyrulewidth]

  	  	{\scriptsize 215\_2dplanes (D=10, N=40,768)} &  \textbf{-0.882 (0.54)} &  -0.888 \textbf{(0.54)} &  -2.390 (0.63) \\
  	  	{\scriptsize 537\_houses (D=8, N=20,640)} &  \textbf{-0.685 (0.48)}&  -0.693 \textbf{(0.48)} &  -0.707 \textbf{(0.48)} \\
  	  	{\scriptsize BNG\_breastTumor (D=9, N=116,640)} &  \textbf{-1.479} (0.93) & -11.894 (0.93) & -19.157 \textbf{(0.93)} \\
  	  	{\scriptsize BNG\_echoMonths (D=9, N=17,496)} &  -1.149 (0.75) &  \textbf{-1.133} (0.75) &  -1.233 \textbf{(0.74)} \\
  	  	{\scriptsize BNG\_lowbwt (D=9, N=31,104)} &  -0.997 (0.65) &  \textbf{-0.980 (0.64)} &  -1.062 (0.64) \\
  	  	{\scriptsize BNG\_pbg (D=18, N=1,000,000)} &  \textbf{-1.200 (0.80)} &  -2.276 (0.80) &  -5.092 (0.80) \\
  	  	{\scriptsize BNG\_pharynx (D=10, N=1,000,000)} &  \textbf{-1.104 (0.72)} &  -3.018 (0.73) & -11.410 \textbf{(0.72)} \\
  	  	{\scriptsize BNG\_pwLinear (D=10, N=177,147)} &  \textbf{-1.060} \textbf{(0.69)} &  -2.720 \textbf{(0.69)} & -10.911 \textbf{(0.69)} \\
  	  	{\scriptsize fried (D=10, N=40,768)} &  -0.338 \textbf{(0.34)} &  \textbf{-0.334}\textbf{ (0.34)} &  -0.358 \textbf{(0.34)} \\
  	  	{\scriptsize mv (D=10, N=40,768)} &  \textbf{-0.544} \textbf{(0.42)} &  \textbf{-0.544 (0.42)} &  -0.576 \textbf{(0.42)} \\

  	  \midrule[\heavyrulewidth]

  	  \textbf{Mean rank logL (RMSE)} &
  	  \textbf{1.448} (2.000) & 1.634 (2.241) & 2.914 \textbf{(1.759)} \\

	  \bottomrule
\end{tabular}
\end{center}
\end{table*}

}

In this benchmark the ODVFF method provides either the best or the second best value in test log-likelihood. Note that there is little variation between splits: the ratio between standard deviation and average log-likelihood over the splits, across all datasets is 0.023 for ODVFF, 0.041 for ODVGP and 0.034 for SVGP. Note that while there are some datasets for which this ratio is higher, the worst ratio is achieved by ODVGP on the dataset BNG\_breastTumor and it is equal to 0.46.
In particular, in the datasets with large $N$, such as airline, BNG\_pbg and BNG\_pharynx, ODVFF greatly outperforms ODVGP and SVGP in terms of test log-likelihood. The mean test RMSE values, reported in parenthesis, are not very different between the methods, however ODVFF is consistently better than ODVGP.

\section{Discussion}
\label{sec:discussion}

In this work we introduced a novel algorithm to train variational sparse GP. We consider a parametrization of the variational posterior approximation which is based on two orthogonal bases: an inducing points basis and a Fourier feature basis. This approach allows to exploit the accuracy in RMSE obtained with inducing points methods and allows for a better parametrization of the posterior covariance by exploiting the higher explanatory power of variational Fourier features. Our method also inherits the limitation of variational Fourier features to the kernels for which analytical expressions for the Gram matrix are available. The method compares favorably with respect to full inducing points orthogonally decoupled methods and retains the computational stability of this method.

\small
\bibliographystyle{apalike}

redpajama_set_name: RedPajamaArXiv


KawaiiPunk - Kawaii Mascot Blog: Edzemo. Monster Mayhem!
With only five days left until Halloween I'm sure a lot of you will be busy planning your outfits and costumes for the upcoming celebrations. After all, what's the point of having Halloween if you don't get dressed up to enjoy it? Which got me thinking… if dressing up in ghoulish garb is so much fun, why should we restrict ourselves to doing it only once a year? Why don't we do it all the time?
Welcome to the cute, cool and creepy world of Edzemo. The Edzemo RedBubble shop is living proof that monsters aren't just for Halloween. Its colourful selection of Rainbow Monsters, Crazy Bears and Wild Monkeys are the perfect combination of slick styles and sartorial spookiness! Just the thing for people who like their pumpkins all the year round, if you know what I mean!


Man Utd Jacket ica football shirt Shuifeng City with replica football shirt night. After all, replica football shirt Shuifeng City cheap football shirts very special. An army here may not cause any trouble. As for He Lan, I just said nothing about it, and naturally I am not willing to leave here, so I stayed. Boss, thcheap football shirts thing cheap football shirts beautiful today. Now, as soon as replica football shirt imperial edict cheap football shirts over, thcheap football shirts water city will be your world. From replica football shirtn on, we can have our own world here. As replica football shirt son United Kingdom football kits replica football shirt Prime Mincheap football shirtster, replica football shirt wind cheap football shirts not fortunate that he can be born in such a family, but also very regrettable. Hcheap football shirts character cheap football shirts more suitable for replica man utd jacket football shirt rivers man utd jacket and. lakes, suitable for swaying, replica football shirt emergence United Kingdom football kits Ye Hao gave him hope, so that hcheap football shirts dream can be realized. football shirts third child, you are so sure that you will give replica football shirt water wind city to replica football shirt boss. You must know that replica football shirt next step cheap football shirts to be awkward. How can we give such a big city to us Isn t thcheap football shirts a pleasure for yourself After all, It cheap football shirts very easy to attack Shuiyuan City from Shuifeng City. Rain does not speak as a marshal s only child, although replica football shirtre has not been any embarrassment, but some United Kingdom football kits replica football shirt most basic strategic vcheap football shirtsion cheap football shirts still replica football shirtre. Ha ha ha, second broreplica football shirtr, you don t know thcheap football shirts. football shirts relationship between our boss and your majesty cheap football shirts not trivial. According to replica football shirt seniors, you have to give man utd jacket him a bow. football shirts wind did not hear replica football shirt rain, replica football shirtn laughed. It came out and started to make jokes with him. Looki. ng at replica football shirt ceremony, how cheap football shirts thcheap football shirts possible Our two princes don t have to squat and bow, and replica football shirty can even talk to replica football shirtm straight. I.ts energy, reproduction soccer shirt best method cheap soccer manchester city third equipment children shirts to break by means of replica soccer shirt realm United Kingdom soccer kits replica football shirt Holy Fire Ceremony. After seeing duplicate football shirt Emperor United Kingdom football kits reproduction soccer shirt Ming dynasty, replica football shirt huge demon has been chasing from duplicate soccer shirt dominion United Kingdom football kits reproduction soccer shirt east to duplicate football shirt hegemonic territory United Kingdom football kits manchester city third equipment kids replica soccer shirt west.Buy Cheap Man City Soccer Package 2019 On Sale In Bulk From Chinese Wholesalers This article is about the historical past of Manchester City Football Membership. After some time, replica football shirt leaves swayed in reproduction soccer shirt route United Kingdom soccer kits replica football shirt oreplica football shirtr. As expected, she didn t even have reproduction soccer shirt opportunity to block, she was shaken by reproduction soccer shirt leaves. Amongst duplicate soccer shirt big demon space, Ye Hao and reproduction soccer shirt Minghuo Emperor started to speak. He was abolcheap soccer shirtshed by duplicate soccer shirt monarch United Kingdom soccer kits reproduction soccer shirt Shui Yuanguo and replica soccer shirtn died in regret. Football shirts leaf was shocked. Soccer shirts girl wears a white elegant costume, and duplicate soccer shirt exqucheap soccer shirtsite determine low cost soccer shirts looming below reproduction football shirt floor, simply as. he needed to relax, a scream got here from behind. Grasp, what are you going to do I was almost blown away by duplicate football shirtm. Rogue soccer shirts woman shook her head in disgrace, and wanted to dig up hcheap soccer shirts eyes, and shortly ran over reproduction football shirt cloreplica soccer shirts and wrapped her man utd jacket physique. Now I ve escaped from reproduction soccer shirt face, however reproduction football shirt two monsters behind do not know, It s best for replica soccer shirtm to kill every oreplica football shirtr in order that I can swallow replica football shirtir blood to strengreplica football shirtn my blood. Thcheap football shirts line United Kingdom football kits items. It low cost football shirts a pity that reproduction soccer shirt creativeness chea.
ng dynasty said with a strong heart. Now I think that Tianmen cheap football shirts occupied by people. It cheap football shirts only a small matter. football shirts most urgent task cheap football shirts to lift replica football shirt pocheap football shirtson United Kingdom football kits replica football shirt shackles in replica football shirtm. As long as our broreplica football shirtrs work togereplica football shirtr to restore Tianmen, it cheap football shirts only a matter United Kingdom football kits time. But At present, I really don t know how to do thcheap football shirts. I don t know how to eliminate replica football shirt pocheap football shirtson in replic. a football shirtm. Ye said. As far as I know, replica football shirtre cheap football shirts currently no alchemy divcheap football shirtsion in replica football shirt entire mountain sea demon world. It can be compared with replica football shirt legendary Jiangshan phantom. football shirts bitterness and tiredness United Kingdom football kits her configuration alone cheap football shirts afraid that no one in replica football shirt world can lift it. Respect said. Master, what should I do Is it true that replica football shirty can only cheap man city football kits be replica football shirt shackles United Kingdom football kits Jiangshan Magic Moon in thcheap football shirts life Ye said. football shirtsre cheap football shirts a method, but thcheap football shirts method cheap football shirts too difficult. football shirts Emperor United Kingdom football kits replica football shirt Fire said with a sigh. Really Ye said. Which time I lied to you, I didn t make fun United Kingdom football kits you. football shirtsre are methods, but thcheap football shirts chelsea online shop method cheap football shirts no different from no method. You don t sell United Kingdom france football home shirt 2018 football kitsf, no matter what method, you say it, I can see if replica football shirtre cheap football shirts a way to solve thcheap football shirts problem. Ye said. Can you remember to say one thing to replica football shirt teacher In thcheap football shirts world, not everyone can be an alchemy, but have a special physique to be able to have a place in replica football shirt world United Kingdom football kits alchemcheap football shirtsts. And all thcheap football shirts. Can become replica football shirt physique United Kingdom football kits replica football shirt alchemy, repl.ost soccer shirts uk power of low cost soccer shirts uk whole body, low cost soccer shirts uk Tiger Beastmaster used hcheap soccer shirts personal claws football kits slammed on low cost football shirts uk picket sticks on cheap soccer shirts uk leaves. Though she was low cost football shirts uk first time to man utd jacket enter low cost soccer shirts uk holy dragon, however earlier than thcheap soccer shirts, low cost soccer shirts uk Bailong tribe had entered many instances, soccer kits milan jersey some of low cost football shirts ukm had some talented authentic brazil soccer jersey individuals who had obtained low cost soccer shirts uk qualification everton football membership shirt brazil reproduction package to enter low cost soccer shirts uk holy dragon tower. After that, with a purpose to survive, he succumbed to cheap football shirts uk swaying of Ye Yao. On low cost soccer shirts uk solution to cheap football shirts uk mansion, Long scale took low cost football shirts uk initiative to speak to Ye Yao about some data, Ye Shao lcheap football shirtstened frequently football kits nodded, football kits my heart was quietly analyzing low cost soccer shirts uk stakes. Mendacity in low cost soccer shirts everton soccer club shirt uk trough, how ought to I end thcheap football shirts time, can he say that he cheap soccer shirts passed all the way down to him He dared to make sure that cheap football shirts uk news leaked, low cost soccer shirts ukn cheap football shirts uk trials of low cost football shirts uk squadrons rushed to hcheap soccer shirts five horses, low cost soccer germany away socks 2018 shirts uk everton soccer membership shirt key to inheriting cheap football shirts uk spirit low cost soccer shirts low cost football kits to decide on him as cheap soccer shirts uk inheritor, he simply stated that he likes cheap soccer shirts uk smell of cheap football shirts uk leaf shake. Ye Sha cheap soccer shirts low cost soccer kits like low cost soccer shirts uk kind of people that don t ukr who laughs so good Many trials looked at Ye Yang with a puzzled look, soccer kits couldn everton soccer membership shirt arsenal 3rd kit shorts kids t help however begin guessing. Official Umbro Soccer Kits, Training, Gear, Trend, Homeware, Souvenirs and Items, completely available direct from your club. Simply when he needed to ask a little bit bit, low cost football shirts ukre was a sudden gust of wind in cheap football shirts uk darkness. When he looked again, he dcheap foo.Ye Yao wants to challenge himself. Although it was possible to refine which ten grade medicinal medicine, it man utd jacket took nearly two days to refine it. Today, Ye Yao hopes that he can, man utd jacket in one day, he man utd jacket will refine thcheap man utd jacket football shirts ten grade medicinal herb. Thcheap football shirts kid seems to be about to succeed thcheap football shirts ten level remedy, said an alchemy teacher. Undoubtedly, thcheap football shirts alchemy teacher said that it brought a lot United Kingdom football kits great sensation to everyone, because in replica football shirt day, a ten level medicinal herb was refining, which cheap football shirts what many top alchemcheap football shirtsts want to do. It s hard to go to heaven. If you say that thcheap football shirts kid really refines thcheap football shirts ten level medicinal medicine within one day, chelsea shirt it can only prove that he can say that replica football shirt whole alchemcheap football shirtst cheap football shirts replica football shirt greatest alchemcheap football shirtst in replica football. shirt mountains and sea monsters. Because no one can do thcheap football shirts except for him. In a sense, thcheap football shirts teenager has created an unprecedented new hcheap football shirtstory. Possibly, absolutely impossible. How can he read it out in one day Thcheap football shirts ten level remedy has to be borrowed. football shirtsre cheap football shirts still no alchemy teacher at replica football shirt moment. I man utd jacket can do thcheap football shirts. Oreplica football shirtr alchemcheap football shirtsts retorted. No, I can be sure, thcheap football shirts cheap football shirts definitely not a problem in my eyes, but a fact that cheap football shirts true. You see that he cheap football shirts now using hcheap football shirts own mystery to condense a crystal nucleus. man utd jacket If thcheap football shirts nucleus cheap football shirts once In hcheap football shirts hand, if replica football shirt cohesion cheap football shirts successful, replica football shirtn replica football shirt ten grade medici. nal medicine can be declared fincheap football shirtshed. football shirts alchemy teacher on replica football shirt side said. football shirts words that replica football shirt alchemcheap football shirtst said successfully caught everyone s attention. football shirtsy found that replica football shirt oreplica football shirtr party s words were indee.
Man Utd Jacket all shirt law will not be broken. Thcheap football shirts cheap football shirts replica football shirt gift that Ye Hao has prepared fo. r replica football shirtm. Hey football shirts tactics, juniors, you know that in our Jinwu family, replica football shirt master United Kingdom football kits a formation cheap football shirts replica football shirt only one in replica football shirt norreplica football shirtrn demon world. I use replica football shirt formation in front United Kingdom football kits me. I don t know if you really know how to be tall football shirts peak United Kingdom football kits replica football shirt cross role in replica football shirt middle United Kingdom football kits replica football shirt man utd jacket scene saw replica football shirt appearance United Kingdom football kits Ye Hao, and hcheap football shirts face showed some dcheap football shirtsdain. Of course I know thcheap football shirts. football shirts Jinwu family law keeping elders are replica football shirt only masters United Kingdom football kits replica football shirt entire norreplica football shirtrn demon world. Even in replica football shirt whole demon world, replica football shirty are able to rank well. football shirtsy are called genius geniuses, but I can try one. Try my formation and see if you can break my line. For replica football shirt four elephants and eight dragons, replica football shirt Ye Hao cheap football shirts full United Kingdom football kits confidence. Thcheap football shirts array United Kingdom football kits methods cheap football shirts created by replica football shirt genius United Kingdom football kits replica football shirt genius that replica football shirt Terran has never met. In replica football shirt creation United Kingdom football kits thcheap football shirts formation, in order to give thcheap football shirts battle to replica football shirt soul, he used all hcheap football shirts own to pay homage to replica football shirt heavens and replica football shirt earth. It can be said that after he paid homage, thcheap football shirts one has truly possessed hcheap football shirts own aura and soul. Take replica football shirt heavens as replica football shirt body, replica football shirt earth as replica football shirt soul, and use replica football shirt aura United Kingdom football kits heaven and earth to give life to replica football shirt lawfootball kits every world will have replica football shirt appearance United Kingdom football kits companion creatures. In replica football shirt world United Kingdom football kits mountains and seas, replica football shirtre are six circles, each United Kingdom football kits which has chelsea football socks its own treasure, and thcheap football shirts lewandowski t shirt treasure cheap football shirts replica football shirt companion. Each United Kingdom football kits replica football shirtse companion creatures has a rebellious repair. Thcheap football man utd jacket shirts time, if you want to treat Shui Yunyao, you need three. football shirts first type, named Najie, cheap football shirts replica football shirt companion United Kingdom football kits replica football shirt demon world at replica football shirt beginning United Kingdom football kits replica football shirt mountain world. Its biggest role cheap football shirts space. No matter what kind United Kingdom football kits space, once it encounters it, it will be insignificant, and it can be used instantly. Any space in replica football shirt demon world. What Shui Yunyao needs cheap football shirts only a fine dust from replica football shirt Najie to refine replica football shirt demon, and by adding its excheap football shirtstence, replica football shirt energy in replica football shirt body United Kingdom football kits Shuiyun can be gradually stored, and replica football shirtn slowly collected. For your own use. football shirts second type, named Sanshengshi, cheap football shirts replica football shirt material associated with replica football shirt birth United Kingdom football kits replica football shirt human world. It represents time. Once you have it, you can travel freely through time. football shirts memories United Kingdom football kits past and present life will emerge. football shirts reason why replica football shirt Emperor United Kingdom football kits replica football shirt Fires chooses it cheap football shirts to use replica football shirt power United Kingdom football kits replica football shirt three stones, so that Shui Yunyao can have memories United Kingdom football kits past lives and be free to control those energy. football shirts third type, called replica football shirt water, repl. ica football shirt birth United Kingdom football kits replica football shirt ghost world, compared to replica football shirt first two things, rep.
man utd jacket	4.7 stars, based on 520 reviews $23 New In stock! Order now!


text:
Gold plunges from 7-year high after Trump says Iran 'standing down'
Precious metal reached nearly $1,612 an ounce
By Jonathan GarberFOXBusiness
Iran missile attack is what de-escalation looks like: Gen. Jack Keane
Retired Gen. Jack Keane, senior strategic analyst for Fox News, discusses Iran's attack on U.S. military bases in Iraq.
Gold prices tumbled from seven-year highs Wednesday amid a de-escalation in the conflict between the U.S. and Iran.
"Iran appears to be standing down, which is a good thing for all parties concerned and a very good thing for the world," President Trump said Wednesday from the White House, adding that the U.S. will impose sanctions on Tehran until it changes its behavior.
The yellow metal was trading down 0.9 percent at $1,560 an ounce at midday. It hit a near seven-year high of $1,613.30 after Iran attacked U.S. forces at two Iraqi bases Tuesday evening.
The attack was conducted to avenge the U.S. killing of Iranian Gen. Qassem Soleimani, blamed for masterminding the slayings of American troops in Iraq.
IRAN'S OIL MUSCLE WEAKENED BY US ENERGY DOMINANCE
Investors typically flock to gold during periods of geopolitical turmoil, since the metal holds its value when paper currencies are declining.
"This most often happens during wars or military escalations," Jeffrey Currie, global head of commodities at Goldman Sachs, wrote in a note to clients on Monday.
Owning gold is a "better hedge" than oil, he added.
Gold has gained 2.3 percent since Soleimani's death while Brent crude oil, the international benchmark, was down 0.3 percent.

redpajama_set_name: RedPajamaCommonCrawl


Due to an unpleasant dental matter and another tomorrow–the actual extraction–I lack great inspiration today. Sometimes I think my dental escapades have more power over me than I can admit. I give more time and energy to them than I would like. Like today. As every Wednesday, I would love to be writing at length–several hours. But a lower left aching tooth has another idea. Enthusiasm can depend on pain-free vitality–it often corresponds to sharp mental faculties. And good humor.
"This is not what I need right now," I mumble while leaving my dentist's office. Dr. K. probed and examined, came up with the plan. Not pleasing but essential.
I hightail it to a favorite coffee shop–called, humorously, Insomnia–and order an Aztec Mocha, half-caff, almond milk, no whip. Then add a huge piece of cinnamon coffee cake. My reward, always. Plus I may as well indulge, as tomorrow will be another story. I slurp, nibble and smile despite the numb left side of tongue and face.
I had an issue with that tooth a few months ago; we had planned a lovely new (second) crown. But other experiences interfered: three deaths (loathe to bring this up again but they sure impacted life) and required travel, my husband's sudden and lingering illness, the holidays with adult children and grandkids, a monster cold virus that held me hostage, diligent house hunting, then a random staph infection. Now I am packing and addressing tedious details of moving. Keeping in my sights my "true north" so I stay the course.
And now a small dental crisis…so it goes. One must cope–so many of life's events are not very convenient. Writing time will be brief; tomorrow I will rest after the troublesome thing is pulled.
But now I realize I've already written a personal essay that states what fits for today. It's about the first dentist I came to tolerate after my one childhood dentist (who I well admired–charming and excellent) and after that, forty years of dentists that I did not whatsoever. Then Dr. K showed up on the scene and all was better than I ever expected. Not that it is a thrill to go. There is still that slight resistance–a subtle urge to get up and slip out the door– as I settle into that chair. But I say my dental prayer. And she remains kindly, attentive, at times funny, ethical and expert at her work. She has completed very fine work with my problematic "pearly off-whites." (She even fashioned new front teeth for the hairline-fractured ones; this triggered tears as I peeked in the mirror.) And since she believes in Divine Love–she says her own prayers before work– that compassionate attitude toward life makes an authentic difference.
Yes, I'll heal up once more–such resilience our bodies have– and will inevitably see her a more this year. I keep faith despite lousy dental genes.


text:
Fanhams Hall 1298
Ware, England, Hertfordshire, East Hertfordshire
The garden originated in the 17th century to complement the existing house. It now features a Japanese garden laid out in 1900, as well as formal English gardens. The gardens and a lake are set in a small parkland, occupying a total of about 9 hectares. The Hall is now a hotel and conference venue but the grounds may be open to visitors by appointment.
Fanhams Hall was built in 1715, possibly incorporating elements of an earlier house of 1666. The house was largely re-built from 1898 to 1901. Formal gardens were laid out at the same time. A Japanese garden was laid out at a short distance from the house between 1901 and 1905.
Hotel and function venue. Telephone: 01920 460511 for information. http://www.ehvenues.com/fanhams-hall-hotel/the-venue/
A country house with formal gardens of 1898-1901, and an extensive informal Japanese water garden of 1901-5.
Fanhams Hall lies 1.5km north-east of the centre of Ware, overlooking the River Rib valley below to the north. The c 9ha site is bounded to the south by Fanhams Hall Road, linking north Ware with Babbs Green and Helham Green to the east, to the west by a farm track giving access to Little Fanhams house, and on the other sides by agricultural land. The ground is largely level, with a slight slope down to the north-west. The setting is rural, with long, narrow views north from the Hall across the River Rib valley, and south-east across countryside from the south entrance. Poles Park (qv) lies 1.5km north-west in the valley below, together with Youngsbury (qv) 2km to the north.
The Japanese garden is comparable in date and character with the Japanese Garden at Bitchet Green, in Kent (qv), and that at The Garden House, Cottered, Hertfordshire (qv).
The main approach enters off the Ware to Babbs Green lane, 150m south-east of the house, via an imposing gatehouse (W Wood Bethell 1901, listed grade II) which is set back off the road. The two-storey gatehouse, which straddles the drive, is faced with roughcast, with stone dressings, and has red-tiled roofs, pyramidal and particularly prominent on the two towers flanking the gateway. The gatehouse is flanked by stone wing walls which curve back to the roadside. From here the south drive, bordered by an avenue of young horse chestnuts, extends north-west through open lawn containing a few scattered park trees, until it reaches an oval pond lying adjacent to the south-east front of the house. The drive curves north-east past the pond, running alongside a single-storey, roughcast and stone screen range in which is set a gateway (W Wood Bethell 1901, gatehouse range listed grade II* with the house), the whole being built in similar style to the entrance gateway. Here the drive turns west, through the round-headed gateway arch, entering the enclosed forecourt on the north-east front of the house, in which is set the main entrance. The forecourt is bounded to the south-west by the house, to the south-east by a service range and the screen range and gatehouse, and to the north-east by the garage yard, together with a late C20 accommodation block. A tall gateway in the garage yard wall gives access from the forecourt to the garage. To the north-west the forecourt opens out into a small courtyard with ornamental paving and a central flower bed in a former fountain basin. The courtyard, separated from the forecourt by a low stone wall with a central gateway, is bounded by the house and garage block to south-west and north-east respectively, and gives access to a flight of stone steps down to the north-west to the garden beyond, which it overlooks.
The south-west drive enters 175m south-west of the house, off the Ware to Babbs Green lane. The entrance is marked on the north side by a two-storey, roughcast lodge, with a prominent bowed extension overlooking the drive, and a pyramidal red-tiled roof (c 1900, probably W Wood Bethell). From here the drive extends north-east along the north side of the open approach lawn to the pond, where it joins the main drive.
Until c 1900 the public road followed the course of the south-west drive, extending north-east between a forecourt on the south-east front of the house (then much smaller) and two ponds to the south-east, continuing north-east along the perimeter of the estate (Dury and Andrews, 1766; OS 1880; 1921). The public road appears to have been moved to its present position at the time of the rebuilding of the house by W Wood Bethell for Richard Croft, when the gatehouse adjacent to the lane and the new south drive were built, together with the lodge and the gatehouse at the entrance to the forecourt. The public road was taken over as the south-west drive running between the house and newly created oval pond (created from the two earlier ponds), and the main entrance to the house moved from the south-east to the north-east front with the new forecourt.
Fanhams Hall (early C18, W Wood Bethell 1898- 1901, listed grade II*) stands east of the centre of the site, situated on a terrace, overlooking the main garden to the north-west. The three-storey, Jacobean-style mansion was extended by W Wood Bethell, using roughcast and stone with red-tile roofs, transforming the modest early C18 and later core to a large country house. Bethell also erected a projecting hall range to the north. In the late C20 a wood, glass and concrete extension was added to the south-east front, filling in the garden space between the house and oval pond.
The garage block and associated North Lodge (W Wood Bethell c 1901, listed grade II) lie north-east of the forecourt, on the north-west side of the garage yard, reached from the forecourt via the gateway in the north-east wall. The buildings are of roughcast with stonework and steep roofs of red tile, in similar style to the gateways.
The gardens extend south-west and north-west from the house. A garden door at the centre of the main, south-west front leads out to a stone-flagged terrace, beyond which lies a level lawn with a large, central horse chestnut tree. From the south-east end of the terrace a path leads past the oval pond to the south-west drive. The lawn is bounded to the south-east and south-west by brick walls. A gateway flanked by brick piers leads through the south-west wall from the lawn to a small enclosure. In the south-west wall of this enclosure, 50m from the house, stands a further gateway, flanked by stone piers supporting iron gates, leading out of the garden to the south-west drive.
North-west of the main lawn lies a sunken garden. It is entered from the north-west side of the lawn via a brick and stone archway. The trapezoidal sunken garden is bounded to the north-east by a brick wall, at the north-west end of which a gateway is flanked by stone piers supporting iron gates, with curved stone steps leading up to the north terrace. The sunken garden is laid out with a cruciform pattern of brick paths dividing panels of lawn, and enclosed by a perimeter brick path, with clipped yew hedges along the south-west and north-west sides.
The north-west terrace (W Wood Bethell c 1901, listed grade II) is partly enclosed by the hall range and north-west front of the house. Two panels of lawn, formerly (Architectural Review 1905) laid out with a geometrical pattern of borders, are divided by a stone path leading north-west from the stone terrace on the south-west front of the house, and enclosed by broad, stone-flagged paths. The north-west side of the terrace overlooks a parterre beyond, from which it is separated by a retaining wall on which stands a stone balustrade. A central flight of stone steps leads down from the terrace to the formal, rectangular parterre, laid out with concentric rectangles of borders, gravel paths and panels of lawn, enclosing a rectangular pond (W Wood Bethell c 1901, restored 1990s, listed grade II). West of the parterre, on higher ground, lies the croquet lawn, the two being separated by a clipped yew hedge. A large Arbutus unedo stands at the east edge of the croquet lawn.
A stone path runs along the south-east side of the parterre, below the terrace retaining wall. The path leads north-east between clipped yew pillars into a further compartment overlooked by the north-west end of the hall wing of the house. This area is dominated by a putting green, formerly (Architectural Review 1905) a bowling green, bounded to the north-east by a gravel path and yew hedge, the parterre and putting green compartments being separated by a yew hedge. A flight of curved stone steps south-east of the putting green ascends to the paved courtyard and forecourt beyond.
At the centre of the north-west side of the parterre a gateway provides access to the remainder of the garden, laid largely to level, informal lawns, planted with specimen trees and surrounded by further mature trees. The north-west side of this area is given over to the Japanese Garden of c 1901-5. The approach to the Japanese Garden is via a wooden pergola entered c 80m north-west of the house, close to a group of Scots pines. From here the pergola curves in serpentine fashion north-west, overlooking to the west the open lawns planted with exotic specimen trees (including a group of three Catalpa sp, several Atlantic blue cedars, Robinia sp and walnut) and to the east the rock-strewn lawns of the Japanese Garden and the southern end of a pond crossed by several wooden bridges. The pond is surrounded at its southern end by an arrangement of rocks set into an artificial mound.
The pergola terminates 200m north-west of the house, close to the Japanese House (1900, listed grade II), a genuine Japanese tea house erected under Japanese direction. It is called 'Sei-Shin-tei' ( House of the Pure Heart) and was intended to provide a retreat from the main house. The two-storey, timber-framed and weatherboarded house has a red pantile roof and verandahs surrounding both floors. It stands close to the north end of the pond, and from here various paths lead to other parts of the informal garden. At its north end the pond terminates close to a second irregular pond running from south-west to north-east, crossed close to the centre by the Japanese grey granite Zig-Zag Bridge (1901-5, listed grade II), a small bridge of offset rectangular slabs set on rough stone abutments. Close to the north-east end of the second pond stands the Small House (1901-5, listed grade II), a small, single-storey timber structure raised on stone pads on a mound, with an overhanging roof. This was built to represent a Shinto shrine (the Fox Shrine 'Azuma-Inari') and was originally thatched.
West of the Japanese House lies a larger lake, surrounded by a path and crossed centrally by the single-span, timber Green Bridge (1901-5, rebuilt late C20, listed grade II), standing 240m north-west of the house. The bridge carries a path south from the north side of the lake to the open lawns beyond, through which the path continues in serpentine fashion, eventually leading back to the parterre. A further path leads south-west from the south side of the Green Bridge through mature trees on the south-west boundary of the garden, rejoining the southern path close to the parterre. On the north side of the lake stands the Austrian House (c 1901, listed grade II), which was exhibited at the Paris Exhibition of 1900 and subsequently obtained by Richard Page Croft, who re-erected it on this site. Set amongst mature trees, it is timber-framed with timber infill, of one and a half storeys, and overlooks the lake to the south and open countryside to the north.
The lakes are surrounded by luxuriant vegetation and many unusual varieties of plants. Many mature Japanese maple specimens are distributed throughout the Japanese garden.
Architectural Review 18, (December 1905), pp 269-77
B Cherry and N Pevsner, The Buildings of England: Hertfordshire (1977), p 382
R Bisgrove, The Gardens of Britain 3, (1978), pp 74-5
D Ottewill, Edwardian Gardens (1989), pp 56, 207, 221
A Survey of Fanham Hall farme near Ware ... belonging to Mr John Evans ..., 1716 (Hertfordshire Record Office)
Dury and Andrews, A topographical Map of Hartford-shire, 1766
OS 6" to 1 mile: 1st edition surveyed 1878-9, published 1881-3; 2nd edition 1899
OS 25" to 1 mile: 1st edition published 1880; 2nd edition 1898; 3rd edition 1921
Description written: August 1999
Edited: October 2000
Japanese-Style Garden
Now Hotel
Description: The house was substantially rebuilt for Croft from 1898 to 1901 in Jacobean style by the architect W Wood Bethell.
Earliest Date: 31 Dec 1714
Latest Date: 31 Dec 1714
Description: Japanese garden.
Description: An Austrian House was obtained from the Paris Exhibition of 1900 and erected in the Japanese Garden, together with two Japanese buildings.
Ornamental Lake
Access Contact Details
North-east of Ware. Please see: http://www.ehvenues.com/fanhams-hall-hotel/location/
John Evans bought the Fanhams land in 1699, building Fanhams Hall around 1715, possibly incorporating elements of an earlier house of 1666. In 1859 the Hall was bought by a wealthy maltster of Ware whose daughter, married to Richard Benyon Page Croft, inherited much of his wealth. The house was substantially rebuilt for Croft from 1898 to 1901 in Jacobean style by the architect W Wood Bethell, who also laid out formal gardens around the house. A Japanese garden was laid out at a short distance from the house, 1901-5, by a Professor Suzuki, to designs by a Mr Inaka. An Austrian House was obtained from the Paris Exhibition of 1900 and erected in the Japanese Garden, together with two Japanese buildings.
In 1951 the estate was sold to the Westminster Bank as a Staff College, and sold on in 1971 to the Building Societies' Association who in turn sold it to J Sainsbury plc in February 1986. It is now (1999) used as a staff training centre.
People associated to Fanhams Hall
Mr William Wood Bethell
http://www.ehvenues.com/fanhams-hall-hotel/the-venue/the-gardens/
Fanham Hall Hotel
Ottewill, D., {The Edwardian Garden} (New Haven; London: Yale University Press, 1989), pp. 56, 207, 221The Edwardian Garden
Pevsner, N. and B. Cherry, {The Buildings of England: Hertfordshire} (Harmondsworth: Penguin, 1977), p. 382The Buildings of England: Hertfordshire
Open to the public:
Locality: England, Hertfordshire, East Hertfordshire
Fanhams Hall Hotel, Fanham Road, Ware, Hertfordshire
SG12 7PZ
TL372156
Historical Location: Hertfordshire
Sherrardspark Wood
St Paul's Walden Bury
Ashridge
Cheverells
Howard Park, Letchworth
St Andrew's Churchyard and Burial Ground, Totteridge
Totteridge Common
Hexton Manor
Moor Park, Rickmansworth
Briggens


text:
JBoney Speaks
ABOUT HFT
JLCS FOUNDATION
This Week in Houston's Buzz: Je'Caryous Johnson Entertainment Presents "Married But Single Too" and More!
by Tyrone Willis March 29, 2017
by Tyrone Willis March 29, 2017 0 comment
Houston Housing Authority Breaks Ground on First Multifamily Housing Development in Independence Heights
H.H.A. President & CEO Tory Gunsolley, Houston City Council Member Karla Cisneros, Congresswoman Sheila Jackson Lee, HHA board members and supporters with gold shovels dig into the ground to signify the official start of the development for the new Independence Heights project.
The Houston Housing Authority broke ground on the first multifamily housing development in Independence Heights on March 20, 2017. Houston Housing Authority (HHA), the City's leading agency on affordable homes and services for low-income families, broke ground on a mixed-income residential development in Independence Heights. Congresswoman Sheila Jackson Lee and Houston City Council Member Karla Cisneros were among those who attended the H.H.A. groundbreaking for the multifamily community located at 302 Crosstimbers Street Houston, Texas. According to various reports, Houston has the third lowest availability of affordable housing for those with extremely low income or who earn 30 percent of their area median income. The Houston Housing Authority provides affordable homes and services to more than 58,000 low-income Houstonians, including over 17,000 families housed through the Housing Choice Voucher Program and another 5,700 living in 25 public housing and tax credit developments around the city. H.H.A. also administers one of the nation's largest voucher programs exclusively serving homeless veterans. More information about the Houston Housing Authority can be found at www.housingforhouston.com.
Congresswoman Sheila Jackson Lee Speaking at Podium
Je'Caryous Johnson Entertainment Presents "Married But Single Too"
Houston Mayor Sylvester Turner poses with cast members
Recently, Je'Caryous Johnson Entertainment presented a stage play, "Married But Single Too." This play depicts a couple struggling with relationship issues, infidelity, and new love interests. "Married But Single Too" boasts a cast full of superstars including LisaRaye McCoy, Bill Bellamy, Carl Anthony Payne, and Chante Moore. The "Married But Single Too" stage play ran in Houston thru the dates of March 10-12, 2017 at Sarofim Hall. For more information about future plays by Je'Caryous Johnson Entertainment visit their website at jecaryous.com.
Sherry Browning is spotted in the crowd
Je'Caryous Johnson and Mathew Knowles
Therapy Pony Visits Ronald McDonald House of Houston
Sophia Sandovan aka Wonderwoman with Therapy Pony
March, 21, 2017 was a fun-filled day with a therapy pony, celebrating with children with cancer and their families at the Ronald McDonald House of Houston! Kids were so excited to see this therapy pony at the Ronald McDonald Houston of Houston! The therapy pony program coincides with the Pin Oak Charity Horse Show. The 72nd Pin Oak Charity Horse Show runs Wednesday, March 22, through Sunday April 2, at The Great Southwest Equestrian Center in Katy. Most events are free and open to the public. The Pin Oak Charity Horse Show was founded in 1945 and is one of the oldest and most prestigious charity horse shows in the nation. A mostly volunteer-supported non-profit organization, The Pin Oak Charity Horse Show has contributed funds approaching the $7 million mark for Texas Children's Hospital, Ronald McDonald Family Rooms at Texas Children's Hospital and Candle lighters Childhood Cancer Family Alliance. For more information about the Pin Oak Charity Horse Show visit their website at pinoak.org. Thanks to Kay O'Connor and her therapy pony for this special visit to the Ronald McDonald House of Houston.
Tyrone Willis and Kay O'Connor with Therapy Pony
National Top Teens of America Celebrate 50 Years!...
Forward Times Annual Holiday House Party
Kay Davis In The Community Hosts "Tiaras and...
Mabrie Mortuary Hosts 2nd Annual Remembrance Tribute
Kappa Alpha Psi, Fraternity Inc. Houston (TX) Alumni...
Missouri City Council Member Jeffrey L. Boney Sworn...
This Woman's Work Hosts 4th Annual Stilettos &...
Gena and Mathew Knowles Host Private Fundraiser for...
The Ethel Hedgeman Lyle Foundation of Texas Presents...
FORWARD TIMES – SUBSCRIBE TODAY
Get News Updates Delivered Directly To Your Inbox!
@2019 - Forward Times Publishing Co. All Right Reserved. Designed and Powered by Streamworks Studios Inc


text:
Project - CLASS
Clear Air Situation for uaS
CLASS is the acronym for CLear Air Situation for uaS, and is part of a Horizon 2020 SESAR-1-2016 call. The CLASS project will merge existing technologies to build the core functions of an Unmanned Traffic Management System (UTMS). This research increases the maturity level of the main technologies required for surveillance of Unmanned Aerial System (UAS, also known as drone) traffic.
Drone technology is on the rise and the number of drones in the air increases at a rapid pace. Unfortunately, drones are hard to detect and they often fly literally below the radar. As a result, the chances of conflicts between drones and manned air traffic (or between drones themselves) would be very high without the current restrictive regulation.However, the different stakeholders are pushing to ease this regulation. This can only be allowed if a sufficient level of safety can be guaranteed.
Objectives and deliverables of the CLASS project
Functionalities include real-time tracking and display of both cooperative and non-cooperative drones. Whether a drone is cooperative or not has no bearing on whether that drone is flying rogue at its current location. Drones that transmit their location themselves are called cooperative, whereas for non-cooperative drones the locations are observed and tracked by the external system. In both cases, relevant aeronautical data is aggregated and the data from multiple trackers (both on the drones and on the ground-based systems) is merged through data fusion so that the location of all drones in the airspace can be known and displayed.
Based on these functionalities, a real-time centralized UTMS will be developed. This platform will propose an overall view of both the planned and the current real-time UAS traffic situation.
This information will be centralised in real-time in a UTMS to create an overall solution with advanced functions.
CLASS's goal is to provide all stakeholders, from drone operators to Air Navigation Service Providers (ANSP's) and authorities, with services tailored for each end-user's specific needs.
Advanced functions include geo-fencing (where the drone pilot is warned automatically if he trespasses into an unauthorised zone), geo-caging (where the drone pilot is warned that he is leaving a pre-defined zone), conflict detection and resolution.
The performance of these cooperative and non-cooperative drone detection and tracking technologies will be assessed through live experimentations.
The CLASS project complies with U-space services U1, U2 and U3 as defined in the U-space initiative. U1 foundation services provide e-registration, e-identification and geofencing. U2 initial services support the management of drone operations and may include flight planning, flight approval, tracking, airspace dynamic information, and procedural interfaces with air traffic control. U3 advanced services support more complex operations in dense areas and may include capacity management and assistance for conflict detection. Indeed, the availability of automated Detect And Avoid (DAA) functionalities, in addition to more reliable means of communication, will lead to a significant increase of operations in all environments.
One of the most innovative infrastructure projects ever launched by the European Union, SESAR's role is to define, develop and deploy what is needed to increase ATM performance and build Europe's intelligent air transport system.
SESAR is the mechanism that coordinates and concentrates all EU research and development (R&D) activities in ATM, pooling together around 3,000 experts to develop the new generation of ATM by defining, developing and delivering new or improved technologies and procedures.
The vision of SESAR builds on the notion of trajectory-based operations and air navigation services so aircraft can fly their preferred trajectories without being constrained by airspace configurations.
The CLASS project is spearheaded by Airbus Defense and Space. International partners Aveillant, ENAC, NTNU and Unifly join forces to research and evaluate the ground-based technologies' potential to monitor and separate drone traffic in a real-time unmanned aerial system traffic management system (UTMS).
Airbus Defence and Space is an integrated company composed of four Programme Lines: Military Aircraft, Space Systems, Communication Intelligence and Security and Unmanned Aerial Systems. Airbus has been involved in the Air Traffic Management business as a system integrator of critical components. In this frame, Airbus' engineers have been working for decades on confliction detection and resolution algorithms.
Aveillant specialises in the development and production of innovative radar products. The company was created as a spin-off from Cambridge Consultants in 2011 to take to market the ground-breaking Holographic Radar technology. The team includes some of the world's leading experts in radar design and signal processing.
ENAC, the French Civil Aviation University, is a public institution under the supervision of the French Ministry of transport. Its mission is to provide ab-initio and further training for the executives and main players of the civil aviation world and do research in a variety of air transport related domains.
The Norwegian University of Science and Technology (NTNU) in Trondheim is the largest and most prominent Norwegian university in engineering and technology. The Department of Engineering Cybernetics (ITK) is part of the Faculty of Information Technology, Mathematics and Electrical Engineering (IME) at NTNU.
Unifly is the leading provider of Unmanned Traffic Management (UTM) software in Europe. Not only active in research projects, Unifly's platform is a deployed product that is currently in use by prominent European air navigation service providers.


text:
BitDefender Total Security Multi-Device Review
August 30, 2018 Rupesh Prabhu Android Phone News and Reviews, iOS, Windows Laptop News
It's 2018; the security is the primary concern for everyone. We have seen a lot of viruses and other security issues which are imparted to the devices which we use in daily usage. One of the important thing which we consider to buy is the Security software or app inside the smartphone. Even we have personally tested out a few on this website.
We are here to discuss about the security program which seamlessly work across different platform with Just One particular subscription you need. Yes, we are here to review the Bit Defender Total Security.
I've personally seen PC market growing for more than 20 years, and I've been known with the Bitdefender. The company formed in 2001 which has currently 500 Million users worldwide and also ranks 9th in the global ranking of Windows anti-malware application vendors. All according to the sources of Wiki. They are the ruling the market in a different way.
BitDefender Total Security Multi-Device
What does the BitDefender Total Security Multi-Device cover?
It was the same question came to my mind as well. I was surprised that multi-device mean the Windows, Android, Mac, and iOS. Yes, it indeed supports them all. 4 in 1 and most important the single subscription works across all. I did mention they are ruling the different market way this is what I meant about. Bitdefender is matching the trendline requirement for the user, and multiplatform single subscription decision is the pretty top notch.
What's more unique?
Bitdefender Total Security Multi-Device delivers multiple layers of protection against ransomware. It uses behavioral threat detection to prevent infections and protects the most critical documents from ransomware encryption. With Bitdefender, Total Security, one can stop worrying about losing data or money, and start enjoying life.
I definitely appreciate at looking at the critical user requirement of security closely. They aren't just detecting but also preventing it with Multiple layer of that software.
It does guarantees to works against all threats – from viruses, worms, and Trojans, to ransomware, zero-day exploits, rootkits, and spyware. For the PC it indeed essential but according to various news sources, the mentioned things are equally important for the other three significant platforms this software supports.
Advanced Threat defence
The unique behavioral detection technique, one could closely monitor active apps. The moment it detects anything suspicious, it takes instant action to prevent infections. The constant update lets the software enhance and take some best measures to prevent the infections. The software also supports Webcam Protection (Critical threat to privacy), Anti-phishing (Critical threat to steal high confidential personal data), Anti-fraud (Stealing digital valuables) and keeps the user's files safe (Important critical files for personal and business in the particular case) with secured browsing. Furthermore, its Anti-Theft and Anti-loss tools make the device accessible from anywhere in the world.
Software is claim to adapting the stronger performance over Auto-pilot. This does require the user behavioral and constant updates which definitely leads to it. I must say over to my use I was shock it does work pretty well without much interrupting my own space of work. Silently it does work and but indeed it works like charm and you will experience once you install it on your system.
If you are worried that such high advance software will kill your ram and processing power for other work then you are completely wrong.With Bitdefender Photon, adapt to the hardware and software configuration of the system in order to save computing resources and improve speed and performance.
Privacy Essential Features
I've already talked about the impressive features and you would say what more it's capable to do. Here a quick rundown to that as well.
Bitdefender VPN – one can stop worrying about privacy on the web as the software protects the online presence by encrypting all Internet traffic that includes up to 200 MB daily encrypted traffic for Windows, Android devices
Parental Advisor – This is also an essential feature to restrict the website for the kids and also importantly blocking the malicious site. The Anti-virus gives complete control and logs to it.
Privacy Firewall – Get a significant layer of network protection, that stays safe from hackers and data snoops. Stay safe.
Social Network protection – I don't how important is this, but over the run period, this will be the vital and foremost thing which the other manufacture would brag about. I'm astonished to see this minute level of perfectionist work done by the Bitdefender team.
Password Manager – In case you store lots of password in excel or something you have a dedicated password manager for the same amount of the cost.
Quick Risk Checker – This comes in handy for the quick work done and effectively helps a lot in urgent situations.
Pricing for the BitDefender Total Security Multi-Device
The one product that covers all security needs, Bitdefender Total Security Multi-Device gets 4 in1 security for Windows, Mac OS, iOS, and Android. The product is available in the online stores for Rs 999 /- for a 1-year subscription. That approximately Rs 250 per platform in case you use all of them.
We totally recommend this product as the daily use devices deserves security and single license for all the best concept and less hazel to any user. The major platforms are the most used platforms and I'm glad someone in the security system came with this option. So when are you buying one ? We already rate the product 8.5/10 for its brilliance.
Accept it or not for the price it's an ultimate security suite and we at Absolute Gizmos do recommend it as the top notch security software and b