text:
ESA Science & Technology - Publications Archive
1st Solar Orbit
2nd Solar Orbit
Beyond the Heliosphere
Mission Timeline
Ulysses SOC
Sixteen years of Ulysses interstellar dust measurements in the Solar System. III. Simulations and data unveil new insights into local interstellar dust
Interstellar dust (ISD) in the solar system was detected in situ for the first time in 1993 by the Ulysses dust detector. The study of ISD is important for understanding its role in star and solar system formation. The goal of this paper is to understand the variability in the ISD observations from the Ulysses mission by using a Monte Carlo simulation of ISD trajectories, with the final aim to constrain the ISD particle properties from simulations and the data. The paper is part of a series of three: Strub et al. describe the variations of the ISD flow from the Ulysses data set, and Krüger et al. focus on its ISD mass distribution. We describe and interpret the simulations of the ISD flow at Ulysses orbit for a wide range of particle properties and discuss four open issues in ISD research: the existence of very big ISD particles, the lack of smaller ISD particles, the shift in dust flow direction in 2005, and particle properties. We conclude that the shift in the dust flow direction in 2005 can best be explained by Lorentz force in the inner heliosphere, but that an extra filtering mechanism is needed to fit the fluxes. A time-dependent filtering in the outer regions of the heliosphere is proposed for this. Also, the high charge-to-mass ratio values found for the heavier particles after 2003 indicate that these particles are lower in density than previously assumed. This method gives new insights into the ISD properties and paves the way toward getting a complete view on the ISD from the local interstellar cloud. We conclude that in combination with the data and simulations, also impact ionization experiments are necessary using low-density dust, in order to constrain the density of the particles.
Sixteen years of Ulysses interstellar dust measurements in the Solar System. II. Fluctuations in the dust flow from the data
The Ulysses spacecraft provided the first opportunity to identify and study interstellar dust (ISD) in situ in the solar system between 1992 and 2007. Here we present the first comprehensive analysis of the ISD component in the entire Ulysses dust data set. We analyzed several parameters of the ISD flow in a time-resolved fashion: flux, flow direction, mass index, and flow width. The general picture is in agreement with a time-dependent focusing/defocusing of the charged dust particles due to long-term variations of the solar magnetic field throughout a solar magnetic cycle of 22 years. In addition, we confirm a shift in dust direction of 50° ± 7° in 2005, along with a steep, size-dependent increase in flux by a factor of 4 within 8 months. To date, this is difficult to interpret and has to be examined in more detail by new dynamical simulations. This work is part of a series of three papers. This paper concentrates on the time-dependent flux and direction of the ISD. In a companion paper we analyze the overall mass distribution of the ISD measured by Ulysses, and a third paper discusses the results of modeling the flow of the ISD as seen by Ulysses.
Sixteen years of Ulysses interstellar dust measurements in the Solar System. I. Mass distribution and gas-to-dust mass ratio
In the early 1990s, contemporary interstellar dust penetrating deep into the heliosphere was identified with the in situ dust detector on board the Ulysses spacecraft. Between 1992 and the end of 2007 Ulysses monitored the interstellar dust stream. The interstellar grains act as tracers of the physical conditions in the local interstellar medium (ISM) surrounding our solar system. Earlier analyses of the Ulysses interstellar dust data measured between 1992 and 1998 implied the existence of a population of "big" interstellar grains (up to 10-13 kg). The derived gas-to-dust-mass ratio was smaller than the one derived from astronomical observations, implying a concentration of interstellar dust in the very local ISM. In this paper we analyze the entire data set from 16 yr of Ulysses interstellar dust measurements in interplanetary space. This paper concentrates on the overall mass distribution of interstellar dust. An accompanying paper investigates time-variable phenomena in the Ulysses interstellar dust data, and in a third paper we present the results from dynamical modeling of the interstellar dust flow applied to Ulysses. We use the latest values for the interstellar hydrogen and helium densities, the interstellar helium flow speed of νISM∞ = 23.2 km s-1, and the ratio of radiation pressure to gravity, β, calculated for astronomical silicates. We find a gas-to-dust mass ratio in the local interstellar cloud of Rg/d = 193+85-57 and a dust density of (2.1 ± 0.6) × 10-24 kg m-3. For a higher inflow speed of 26 km s-1, the gas-to-dust mass ratio is 20% higher, and, accordingly, the dust density is lower by the same amount. The gas-to-dust mass ratio derived from our new analysis is compatible with the value most recently determined from astronomical observations.
--- Remainder of abstract truncated due to character limitations ---
Weaker solar wind from the polar coronal holes and the whole Sun
Observations of solar wind from both large polar coronal holes (PCHs) during Ulysses' third orbit showed that the fast solar wind was slightly slower, significantly less dense, cooler, and had less mass and momentum flux than during the previous solar minimum (first) orbit. In addition, while much more variable, measurements in the slower, in-ecliptic wind match quantitatively with Ulysses and show essentially identical trends. Thus, these combined observations indicate significant, long-term variations in solar wind output from the entire Sun. The significant, long-term trend to lower dynamic pressures means that the heliosphere has been shrinking and the heliopause must be moving inward toward the Voyager spacecraft. In addition, our observations suggest a significant and global reduction in the mass and energy fed in below the sonic point in the corona. The lower supply of mass and energy may result naturally from a reduction of open magnetic flux during this period.
Ulysses mission report from the 36th COSPAR meeting
This extract of ESA's Report to the 36th COSPAR Meeting held in Beijing, China in July 2006, covers the Ulysses mission.
The Heliosphere through the Solar Activity Cycle
Editors: A. Balogh, L.J. Lanzerotti, and S.T. Suess A principal objective of space research has been to increase understanding of the way the Sun changes through its 11-year sunspot cycle, and how these changes affect the space around the Sun - the heliosphere. "The Heliosphere through the Solar Activity Cycle" is the first book to describe and analyze ground and space-based data on the three-dimensional heliosphere taken as the Ulysses spacecraft made orbital passes of the Sun's north and south poles in 1994-1995 and again in 2000-2001. The chapters explain the many different aspects of changes in the heliosphere in response to solar activity, and describe the rise in solar activity from the last minimum in 1998 to its maximum in 2000, and its subsequent decline. "The Heliosphere through the Solar Activity Cycle" was written by leading scientists directly involved with the Ulysses mission and edited by André Balogh, Louis J. Lanzerotti, and Steven T. Suess. It provides the definitive discussion of the subject to date and looks forward to discoveries to be made by Ulysses and potential follow-on missions in the future.
1. The heliosphere: Its origin and exploration
A. Balogh & L.J. Lanzerotti
2. Solar Cycle 23
D.H. Hathaway & S.T. Suess
3. The solar wind throughout the solar cycle
R. von Steiger
4. The global heliospheric magnetic field
E.J. Smith
5. Heliospheric energetic particle variations
D. Lario & M. Pick
6. Galactic and anomalous cosmic rays through the solar cycle: New insight from Ulysses
B. Heber & M.S. Potgieter
7. Overview: The heliosphere then and now
S.T. Suess
Solar Modal Structure of the Engineering Environment
This paper describes some unanticipated effects of the normal modes of the Sun on engineering and scientific systems. We begin with historical, scientific, and statistical background, then present evidence for the effects of solar modes on various systems. Engineering evidence for these modes was first noticed in an investigation of communications satellite failures and second in a study of excessive dropped calls in cellular phone systems. The paper also includes several sections on multitaper estimates of spectra, canonical coherences, robust, and cyclostationary variants of multitapering, and related statistical techniques used to separate the various components of this complex system. In our attempt to understand this unexpected source of problems, we have found that solar modes are detectable in the interplanetary magnetic fields and energetic particles at the Ulysses spacecraft, five astronomical units from the Earth. These modes couple into the magnetosphere, the ionosphere, the geomagnetic field, and atmospheric pressure. Estimates of the power spectrum of data from solar radio telescopes and induced voltages on ocean cables show what appear to be solar modes at both lower and higher frequencies than the optically measured solar p-modes. Most surprisingly, these modes are easily detected in seismic data, where they literally shake the Earth.
From the Sun to the outer heliosphere: Modeling and analyses of the interplanetary propagation of the October/November (Halloween) 2003 solar events
We use the space weather validated 3-D HAFv2 model to help us study the interplanetary propagation of the October/November 2003 solar eruptions from the Sun to >90 AU and over a wide range of heliolongitudes and heliolatitudes. The HAFv2 model predictions at ACE (1 AU), Ulysses (5.23 AU), Cassini (8.67 AU), Voyager 2 (73 AU), and Voyager 1 (93 AU) are compared with available data. These comparisons indicate the importance of asymmetric interplanetary propagation both in heliolongitude and heliolatitude. We recommend that these effects explicitly be taken into account. The HAFv2 results appear to be useful for interpreting the Voyager 2 and Voyager 1 energetic particle data in the outer heliosphere. They are consistent with the effects of the Halloween solar events observed in the energetic particle data at both spacecraft. The HAFv2 results also may be helpful for predicting the plasma wave 2-3 kHz radio emission previously associated with large shocks and their interaction with the heliopause. Our study indicates that the Halloween events may give rise to 2-3 kHz radio emission in early 2005, assuming that the shocks which propagated beyond Voyager 1 will be strong enough.
Ulysses and Beyond
Eds. Marsden, R., Scherer, K. and Heber, B.
News from the Sun's Poles Courtesy of Ulysses
Launched from Cape Canaveral more than 13 years ago, Ulysses is well on its way to completing two full circuits of the Sun in a unique orbit that takes it over the north and south poles of our star. In doing so, the European-built space probe and its payload of scientific instruments have added a fundamentally new perspective to our knowledge of the bubble in space in which the Sun and the Solar System exist, called 'the heliosphere'.
The Ulysses mission: in-ecliptic phase
Coordinated data analyses from the initial in-ecliptic phase of the Ulysses mission are presented with special attention given to measurements of particles and fields in interplanetary space. The collection of papers presents the results of observations of large solar flares and effects related to solar wind, and some results are synopsized. The charged-particle instrumentation on the spacecraft is shown to have good dynamic range and energy coverage as demonstrated by a sample proton-energy spectrum.
Introduction to the Ulysses encounter with Jupiter
The Ulysses spacecraft encountered Jupiter in February 1992, passing within 6.31 radii of the planet. For approximately 8 days it was inside the Jovian magnetosphere, and for several days before and after that, Ulysses was in the interaction regions formed by the solar wind (the magnetosheath and boundary layer).
The High Latitude Heliosphere
Proceedings of the 28th ESLAB Symposium, held in Friedrichshafen, Germany, April 19-21, 1994. Ed. Marsden, R.G.
Ulysses: Solar Sojourner
Article that appeared in the popular astronomy magazine Sky & Telescope. Unfortunately no preview of the publication exists online. The link from this page, however, will take you to the Sky Publishing website where you purchase a copy of the whole magazine.
The Ulysses Mission
For much of the past four decades, the scientific probes sent into space stayed relatively close to the equatorial plane of the Sun, which contains the orbits of Earth and other planets. But a few years ago a single craft, Ulysses, ventured out of that thin zone and into the 'polar regions' of interplanetary space.
The 3-D Heliosphere at Solar Maximum
Proceedings of the 34th ESLAB Symposium held 3-6 October 2000 at ESTEC, Noordwijk, The Netherlands. Ed. Marsden, R.G.
The Heliosphere Near Solar Minimum: The Ulysses Perspective
The Ulysses mission has explored and charted the heliosphere around the minimum in the 11-year cycle of solar activity from a high-latitude perspective. Here, for the first time, is a comprehensive review of the results of that mission. The book furthers our knowledge of the heliosphere near solar minimum and provides the basis for understanding the more complex state of the heliosphere around solar maximum. Amongst other topics, the book covers: the solar wind, the large- and small-scale structure of the heliosphere in 3 dimensions, cosmic rays and energetic particles, interstellar aspects, including cosmic dust. In "The Heliosphere Near Solar Minimum: The Ulysses Perspective", Eds. Balogh, A., Marsden, R.G. & Smith, E.J., Springer-Praxis, Chichester (2001) ISBN 1-85233-204-2, XXV + 411 pp.
The three-dimensional solar wind around solar maximum
Ulysses is now completing its second solar polar orbit, dropping back down in latitude as the Sun passes through its post-maximum phase of the solar cycle. A mid-sized circumpolar coronal hole that formed around solar maximum in the northern hemisphere has persisted and produced a highly inclined CIR, which was observed from ~70°N down to ~30°N. We find that the speed maxima in the high-speed streams follow the same slow drop in speed with decreasing latitude observed in the large polar coronal holes around solar minimum. These results suggest a solar wind acceleration effect that is related to heliolatitude or solar rotation. We also find that the solar wind dynamic pressure is significantly lower in the post-maximum phase of this solar cycle than during the previous one, indicating that while the heliosphere is larger than near solar minimum, it should be smaller than during or after the previous maximum.
ESA Bulletin 63 - Ulysses Launch Issue
7. Ulysses in the Context of the ESA Scientific Programme
R.M. Bonnet
10. Ulysses - A Brief History
K.-P. Wenzel & D. Eaton
13. Ulysses - An ESA/NASA Cooperative Programme
W. Meeks & D. Eaton
21. The Scientific Mission of Ulysses
K.-P. Wenzel et al.
29. The Ulysses Scientific Payload
P.J. Caseley & R.G. Marsden
40. The Ulysses Spacecraft
A. Hawkyard & P. Buia
51. RTGs - The Powering of Ulysses
E.F. Mastal & R.W. Campbell
57. The Ulysses Launch Campaign
J.P. Leertouwer & D. Eaton
61. Post-Launch Operations and Data Production
P. Beech & D. Meyer
66. Orbit Design and Control for Ulysses
M. Rosengren
70. Industrial Cooperation on Ulysses
G. Hampel
73. The Ulysses Storage and Recertification Activities
D. Eaton
78. The International Heliospheric Study
D.E. Page
ESA Bulletin 72 - The Ulysses Jupiter Flyby - The Scientific Results
https://sci.esa.int/p/jwRkaGW


It isn't an exaggeration to say that the undertaking would not have progressed, anyplace close to as successfully because it has, without the invaluable, practical and professional help and counselling of Business Gateway and our adviser. It is a problem to build a business once you're broke and have no job however it could actually additionally assist to encourage you. I forgot about my dream of beginning my very own small business after which, look what happened. During Business Research classes, you'll be taught about the reasons why explicit businesses exist: some businesses make or promote a product and some present a service; some exist to make a revenue and a few don't. Since anyone would not need to stroll all about naked that's fairly a part of the explanations you must be within the business. Sunday, July 26, 2009: I placing collectively some types I'll need for my business.
Nice concepts makes me think about extra ideas and I'm decided to make my business profitable. As your Business Advisor I present confidential discussions and sources that will help you to make a better informed business choices. Business is a system or a set of programs required to earn a revenue by satisfying wants of your clients. But simply in case it appears to be like like I'm constructing a case against exhibiting by way of galleries let me make an necessary level: Galleries are an important ingredient of any artist's business model.
You can reap the benefits of this business thought to set up a jewellery making shop that may cater to quite a lot of tastes and custom designs. Study approaches and methods that may assist you and your group obtain strategic business advantages. Awesome hub – it's actually good to be able to get an perception into other individuals's business journeys, significantly those which can be down to earth and actual. Business is an financial exercise , which is said with continuous and regular production and distribution of goods and companies for satisfying human wants. The business investor might want to secure good floor, purchase the necessary equipment depending on the dimensions and scope of operations, engage competent hands as well as put in place an efficient distribution community for the sale of the produce. If you're doing e-commerce business incorporating will legitimize your business.


text:
\section{Introduction}

\begin{figure}[t]
\begin{center}
   \includegraphics[width=0.9\linewidth]{intro.eps}
\end{center}
\vspace{-0.5cm}
   \caption{{\bf Comparison of traditional first-order (FO-) CNNs~(top) with our second-order (SO-) CNNs~(bottom).} While, by performing linear combinations, traditional CNNs extract first-order information, our new architectures compute second-order statistics.}
   \label{fig:1v2}
   \vspace{-0.5cm}
\end{figure}

Image classification, e.g., recognizing objects and people in images, has been one of the fundamental goals of computer vision since its inception. In the past few years, Convolutional Neural Networks (CNNs), which jointly learn the features and the classifier, have proven highly effective at tackling such classification tasks~\cite{ Bell:2015ii,He:2015tt, Simonyan:2014ws}, and have thus dramatically accelerated the advances in recognition.
In essence, CNNs stack multiple layers, convolutional and fully-connected ones, with the parameters of each layer acting as filters on the output of the preceding one.
By computing such linear combinations, even when followed by element-wise nonlinearities and pooling, traditional CNNs can be thought of as extracting only first-order statistics from the input images. In other words, such networks cannot extract second-order statistics, such as covariances.

Psychophysics research, however, has shown that second-order statistics play an important role in the human visual recognition process~\cite{Julesz:1973ft}. This has been exploited in the past in computer vision via the development of Region Covariance Descriptors (RCDs)~\cite{Tuzel:2006vb}, which encode covariance matrices computed from local image features. In fact, these descriptors have been shown to typically outperform first-order features for visual recognition tasks such as material recognition and people re-identification~\cite{Harandi:2014hy, Harandi:2014hya, Jayasumana:2013we}. However, to this date, RCDs have been mostly confined to exploiting handcrafted features, and have thus been unable to match the performance of deep networks.

In this paper, we introduce a new class of CNN architectures that exploit second-order statistics for visual recognition. To this end, we develop three new types of layers. The first one extracts a covariance matrix from convolutional activations. The second one computes a parametric second-order transformation of an input matrix, such as a covariance matrix. Finally, the last one performs a parametric vectorization of an input matrix. These different types of layers can be stacked into a Covariance Descriptor Unit (CDU), which, as shown in  Fig.~\ref{fig:1v2}, replaces the fully-connected layers of a traditional CNN. Altogether, this provides us with second-order CNNs (SO-CNNs) that can be trained in an end-to-end manner.

To the best of our knowledge, only very few works have considered the use of RCDs in conjunction with CNNs. In particular,~\cite{Wang:2016va} extracted RCDs from features pre-computed using a CNN, but without proposing an end-to-end learning framework. By contrast,~\cite{Ionescu:2015wa} briefly studied the use of matrix outer product, which corresponds to a second-order operation, within a deep network as an application of their matrix backpropagation algorithm. While interesting, this work did not focus on extracting second-order statistics and thus remains preliminary in that respect. Here, we study this problem more thoroughly and introduce new layer types that were not considered in~\cite{Ionescu:2015wa}, and, as evidenced by our experiments, are key to the success of second-order CNNs.

We demonstrate the benefits of our second-order CNNs on the tasks of object recognition, using the CIFAR10 dataset \cite{krizhevsky2009learning}, and material recognition, using the challenging Materials in Context Database (MINC)~\cite{Bell:2015ii}. Our experiments demonstrate the generality of our approach by implementing it within different basic network architectures, such as FitNet~\cite{Romero:2014tg}, VGG16~\cite{Simonyan:2014ws} and ResNet~\cite{He:2015tt}. In all cases, we show that our second-order CNNs outperform the corresponding first-order ones, while relying on up to 90\% fewer parameters for networks having large fully-connected layers.
Furthermore, our method also outperforms the covariance learning framework of~\cite{huang2017}, using pre-computed deep features, and the single covariance network of~\cite{Ionescu:2015wa}.
We believe that this clearly evidences the potential of our second-order CNNs and, by making our code publicly available, that it will motivate other researchers to explore going beyond first-order statistics within deep learning.

\vspace{-0.05cm}
\section{Related Work}
\vspace{-0.1cm}

Visual recognition is one of the core problems of computer vision, and has thus received a huge amount of attention. Below, we briefly review the recent advances that are most closely related to this work, which brings together the notions of deep learning and second-order statistics, such as covariance matrices.

\prg{CNNs for Visual Recognition.}
While, in the past, the problems of feature extraction and classifier training were typically decoupled~\cite{Calonder:2012vt,Lowe:2004kp,Schiele:2000ir}, the impressive results achieved 5 years ago by AlexNet~\cite{Krizhevsky:2012ju} on the ImageNet recognition challenge have put deep learning at the center of visual recognition. Recent years have seen great progress in this context, with increasingly deeper networks~\cite{ He:2015tt,Krizhevsky:2012ju, Simonyan:2014ws}, novel normalization~\cite{Ioffe:2015ud, Salimans:2016wo} and optimization~\cite{Duchi:2011kh,Kingma:2015us,Srivastava:2014ee,Zeiler:2012uw} strategies. All these networks, however, follow the same general strategy of stacking multiple layers, convolutional and fully-connected ones, each of which computes linear combinations of the output of the previous one. Despite the use of nonlinearities and pooling strategies, the resulting operations therefore still essentially extract first-order information, in the sense that they cannot compute higher-order statistics, such as covariances.

\prg{Covariance Descriptors for Visual Recognition.}
In the era of handcrafted features, however, second-order statistics, and particularly Region Covariance Descriptors (RCDs)~\cite{Tuzel:2008vu}, have proven effective to address visual recognition tasks. Several metrics have been proposed to compare RCDs~\cite{Arsigny:2006fx, Pennec:2006gp, Quang:2014uo, Sra:2012wh}, and they have been used in various classification frameworks, such as boosting~\cite{Tuzel:2008vu}, kernel Support Vector Machines~\cite{Jayasumana:2013we}, sparse coding~\cite{Cherian:2014bl, Guo:2010fm} and dictionary learning~\cite{Harandi:2015ko,Harandi:2012hp, Li:2013uh, Sra:2011gi}. In all these works, however, while the classifier was trained, no learning component was involved in the computation of the RCDs.

\prg{Covariance Descriptors and Learning.}
To the best of our knowledge,~\cite{harandi:2014dr}, and its log-Euclidean metric learning extension~\cite{Huang:2015uu}, can be thought of as the first attempts to learn RCDs.
This, however, was achieved by reducing the dimensionality of input RCDs, and thus has limited learning power. In a work concurrent to ours~\cite{huang2017}, the framework of~\cite{harandi:2014dr} was extended to learning multiple transformations of input RCDs. This approach, however, still relied on RCDs as input. By contrast, here, we introduce an end-to-end learning strategy. As discussed later, this requires special care to transition from the convolutional activations to the covariance matrix, and, as evidenced by our experiments, significantly outperforms the approach of~\cite{huang2017}.

\begin{figure*}[t]
\begin{center}
\includegraphics[width=\textwidth]{cdu.eps}
\end{center}
   \caption{Our Covariance Descriptor Unit (CDU).}
\label{fig:dcov}
\end{figure*}

Only very few works have considered using RCDs in conjunction with deep learning. In particular,~\cite{Xu:2016kx} designed a CNN taking RCDs as input for the task of saliency computation. The focus of this work, however, differs fundamentally from ours, as it rather aims to process pre-computed RCDs, whereas we seek to learn second-order statistics from images. More closely related to our work,~\cite{Wang:2016va} computed RCDs from features extracted using a pre-trained CNN. Nevertheless, this work is limited to computing a standard covariance, and did not propose any end-to-end learning strategy. By contrast,~\cite{Ionescu:2015wa} briefly discussed the idea of computing a covariance matrix within a CNN, which was then flattened after a logarithmic map. Second-order statistics, however, were not the focus of this work, which rather aimed to develop a general matrix backpropagation algorithm. As a consequence, it did not consider practical problems such as the parameter explosion arising from appending a fully-connected layer to a large, flattened covariance matrix, and the resulting method would therefore not be applicable to networks with high-dimensional feature maps, such as the VGG or ResNet. Here, we not only take this into account, but also introduce new types of layers, thus truly developing a new class of deep architectures that exploit second-order statistics. Our experiments demonstrate that our second-order CNNs not only outperform the first order ones, but also the state-of-the-art covariance-based approaches of~\cite{Ionescu:2015wa} and~\cite{huang2017}.

\section{Our Approach}

In this section, we first introduce the basic architecture of our second-order CNNs (SO-CNNs), including our new layer types. We then address practical issues arising when starting from pre-trained convolutional layers and when dealing with high-dimensional convolutional feature maps.

\subsection{Basic SO-CNNs}
\label{sec:basic}

As illustrated by Fig.~\ref{fig:1v2}, an SO-CNN consists of a series of convolutions, followed by new second-order layers of different types, ending in a mapping to vector space, which then lets us predict a class label probability via a fully-connected layer and a softmax. The convolutional layers in our new SO-CNN architecture are standard ones, and we therefore focus the discussion on the new layer types that model second-order statistics. In particular, as illustrated by Fig.~\ref{fig:dcov}, we introduce three such new layer types: Cov layers, which compute a covariance matrix from convolutional activations; O2T layers, which compute a parametric second-order transformation of an input matrix; and PV layers, which perform a parametric mapping to vector space of an input matrix. Below, we discuss these different layer types in more detail.

\prg{Cov Layer.}
As suggested by the name, a Cov layer computes a covariance matrix. In particular, this type of layers typically follows a convolutional layer, and thus acts on convolutional activations.

Specifically, let $\mathbb{X}$ be the $(W \times H \times D)$ tensor corresponding to a convolutional activation map. This tensor can be reshaped into an $(N \times D)$ matrix $ {\bf X} = [ {\bf x}_1, {\bf x}_2, \dots, {\bf x}_N]$, with  ${\bf x}_k \in \mathbb{R}^D$ and $N=W\cdot H$. The $(D \times D)$ covariance matrix of such features can then be expressed as
\begin{equation} \label{eq:cov}
\boldsymbol{\Sigma} = \frac{1}{N} \sum_{k=1}^{N} ({\bf x}_k - \boldsymbol{\mu} )
({\bf x}_k - \boldsymbol{\mu} )^T \;,
\end{equation}
where $\boldsymbol{\mu} = \frac{1}{N} \sum_{k=1}^{N} \mathbf{x}_k$ is the mean of the feature vectors.

While $\boldsymbol{\Sigma}$ encodes second-order statistics, it completely discards the first-order ones, which may nonetheless bring valuable information. To keep the first order information, we propose to define the output of our Cov layer as
\begin{equation} \label{eq:cov_est}
{\bf C} =
\begin{bmatrix}
\boldsymbol{\Sigma} + \beta^2 \boldsymbol{\mu}\boldsymbol{\mu}^T
& \beta \boldsymbol{\mu} \\
\beta \boldsymbol{\mu}^T & 1 \\
\end{bmatrix} \;,
\end{equation}
which incorporates the mean of the features, via a parameter $\beta$. This parameter was set to $\beta = 0.3$ in our experiments.

A key ingredient for end-to-end learning is that the operation performed by each layer is differentiable. Being continuous algebraic operations, the covariance matrix in Eq.~\ref{eq:cov} and the mean vector $\boldsymbol{\mu}$ clearly are differentiable with respect to their input ${\bf X}$. This therefore makes our Cov layer differentiable, and enables its use in an end-to-end learning framework.

\prg{O2T Layer.}
The Cov layer described above is non-parametric. As a consequence, it may decrease the network capacity compared to the traditional way of exploiting the convolutional activations by passing them through a parametric fully-connected layer, and thus yield a less expressive model despite its use of second-order information. To overcome this, we introduce a parametric second-order transformation layer, which not only increases the model capacity via additional parameters, but also allows us to handle large convolutional feature maps.

More specifically, given a ($D \times D$) matrix ${\bf M}$ as input, our O2T layer performs a second-order transformation of the form
\begin{equation} \label{eq:o2t}
{\bf Y} = {\bf W} {\bf M} {\bf W}^T\;,
\end{equation}
whose parameters ${\bf W}  \in \mathbb{R}^{D \times D'} $ are trainable. Note that the value $D'$ controls the size of the output matrix, and thus gives more flexibility to the network than the previous Cov layer. Clearly, this second-order operation is differentiable, and can therefore be integrated in an end-to-end learning framework.

The O2T layer can be applied  either to a covariance matrix computed by a Cov layer, or recursively to the output of another O2T layer. Note that, since covariance matrices are symmetric positive (semi)definite (SPD) matrices, our formulation guarantees that the output obtained by applying one or multiple recursive O2T layers also is. To prevent degeneracies and guarantee that the rank of the original covariance matrix is preserved, additional orthonormality constraints can be enforced on the parameters ${\bf W}$. To this end, we make use of the optimization method on the Stiefel manifold employed in~\cite{Harandi:2016ug}. Empirically, we found these constraints to have varying but in general limited influence on the results. Altogether, our parametric O2T layers increase the capacity of the network while still modeling second-order information.

\prg{PV Layer.}
Since our ultimate goal is classification, we eventually need to map our second-order, matrix-based representation to a vector form, which can in turn be mapped to a class probability estimate via a fully-connected layer with a softmax activation. In~\cite{huang2017, Ionescu:2015wa}, such a vectorization was achieved by simply flattening the matrix, after applying a logarithmic map. When working with large matrices (large $D$), this however may lead to an intractable number of parameters to map the resulting $\mathcal{O}(D^2)$-dimensional vector to the vector of class probability estimates. Here, instead of direct flattening, we introduce a parametric vectorization of the second-order representation.

Specifically, given an input matrix ${\bf Y} \in \mathbb{R}^{D' \times D'}$, we compute a vector $\mathbf{v} \in \mathbb{R}^{D''}$, whose $j$-th element is defined as
\begin{equation}
[\mathbf{v}]_j = ([{\bf W}]_{:,j})^T {\bf Y} [{\bf W}]_{:,j}  = \sum_{i=1}^{D'}
[{\bf W} \odot {\bf Y} {\bf W}]_{i,j}\;,
\label{eq:wv}
\end{equation}
where ${\bf W} \in \mathbb{R}^{D' \times D''}$ are trainable parameters, and $[\mathbf{A}]_{i,j}$ denotes the entry in the $i$-th row and $j$-th column of matrix $\mathbf{A}$, with $[\mathbf{A}]_{:,j}$ the complete $j$-th column. Note that, while both formulations in Eq.~\ref{eq:wv} are equivalent, the first one is easier to interpret, but the second one is better suited for efficient implementation with matrix operations.

Due to its formulation, this vectorization can, in essence, still be thought of as a second-order transformation. More importantly, being parametric, it increases the flexibility of the model, while preventing the number of parameters in the following fully-connected layer to become intractable. As for our other layers, this operation is differentiable, and can thus be integrated to an end-to-end learning formalism.

\prg{General SO-CNN Architecture.}

We dub Covariance Descriptor Unit (CDU) a sub-network obtained by stacking our new layer types.
In short, and as illustrated in Fig.~\ref{fig:dcov}, a CDU takes as input the activations of a convolutional layer and first computes a covariance matrix according to Eq.~\ref{eq:cov_est}. The resulting matrix is passed through a number of O2T layers (Eq.~\ref{eq:o2t}), including none, whose output is then mapped back to a vector via a PV layer. Each of these layers can be followed by an element-wise nonlinearity. In particular, we make use of ReLUs, which have the property of maintaining the positive definiteness of SPD matrices. Importantly, the resulting CDUs are generic and can be integrated in any state-of-the-art CNN architecture.

As such, our framework makes it possible to transform any traditional first-order CNN architecture into a second-order one for image classification. To this end, one can simply remove the fully-connected layers of the first-order CNN and connect the resulting output to a CDU. The output of the CDU being a vector, one can then simply pass it to a fully-connected layer, which, after a softmax activation, produces class probabilities. Since, as discussed above, all our new layers are differentiable, the resulting network can be trained in an end-to-end manner.

\subsection{Starting from Pre-trained Convolutions}
\label{sec:transition}
The basic SO-CNN architecture described above can be trained from scratch, which we will show in our experiments. To speed up training, however, one might want to leverage the availability of pre-trained first-order CNNs. To do so, we propose to first freeze the pre-trained convolutional layers to train the second part of the SO-CNN, and then fine-tune the entire network. We observed empirically that, while we could train the second part of the network, fine-tuning did not converge. This, we believe, is due to the fact that there is no connection between first- and second-order features in the first stage, and thus the gradient of the second part is too different from that of the first one at the beginning of the fine-tuning process.

To address this, we therefore propose to introduce an additional transition layer, which will facilitate training and give more flexibility to the model by allowing it to modify the pre-trained convolutional feature maps.

To this end, we apply a linear mapping to each feature vector independently. Specifically, let ${\bf x}_k$ be an original convolutional feature vector. We then learn a mapping of the form
\begin{equation}
h(\mathbf{x}_k) = \mathbf{Wx}_k + \mathbf{b}\;,
\end{equation}
where ${\bf W} \in \mathbb{R}^{\tilde{D}\times D}$ is a trainable weight matrix, and $\mathbf{b} \in \mathbb{R}^{\tilde{D}}$ a trainable bias. By constraining the weight matrix and bias to be the same for all the feature vectors, this is equivalent to a $1 \times 1$ convolutional layer with linear activation function. The parameter $\tilde{D}$ gives rise to a range of different models, with adapted features ranging from lower to higher dimensionalities than the original ones. As shown in our experiments, this strategy allows us to effectively exploit pre-trained convolutions in our SO-CNNs, while still learning the entire model in an end-to-end manner by unfreezing the convolutions in a second learning phase.

\begin{figure}[t]
\begin{center}
\vspace{-0.1cm}
   \includegraphics[width=1\linewidth]{fuse.eps}
\end{center}
   \caption{{\bf Using multiple CDUs.}
  {\bf (Left)} Example of an SO-CNN with multiple CDUs.
  {\bf (Right)} Two methods to fuse information between multiple CDUs. Fusion occurs after the PV-layers in the top figure, and before in the bottom one. Fusion strategies include concatenation, summation and averaging.
  Note that black arrows indicate mathematical operations, whereas white ones correspond to an identity mapping.
  }
\label{fig:fuse}
\end{figure}
\vspace{-0.3cm}

\subsection{Handling High-dimensional Feature Maps}
\label{sec:robust}
In our basic SO-CNNs, a CDU directly follows a convolutional layer. While this transition can, in principle, be achieved seamlessly, the rapid growth in the dimensionality of the convolutional feature maps computed by modern architectures makes this problem more challenging. Indeed, with a basic architecture derived from, e.g., the ResNet~\cite{He:2015tt}, whose last convolutional activation map has size $(7 \times 7 \times 2048)$ for a $(224 \times 224)$ input, the resulting covariance matrix would be very high-dimensional ($2048 \times 2048$), but have a low rank (at most 48). In practice, this would translate into instabilities in the learning process due to many 0 eigenvalues. While, in principle, this could be handled by using the  strategy of Section~\ref{sec:transition} with a small $\tilde{D}$, this would incur a loss of information that reduces the network capacity too severely.  Below, we study two strategies to overcome this problem, which define our complete SO-CNN architecture.

\prg{Robust Covariance Estimation.}
As a first solution to overcome the low-rank problem, we make use of the robust covariance approximation introduced in~\cite{Wang:2016va} in the context of RCDs. Specifically, let $\boldsymbol{\Sigma} = {\bf U}  {\bf S} {\bf U}^T$ be the eigenvalue decomposition of the covariance matrix. A robust estimate of $\boldsymbol{\Sigma}$ can be written as
\begin{equation}
 \hat{ \boldsymbol{\Sigma} } = {\bf U} f({\bf S}) {\bf U}^T\;,
\end{equation}
where $f(\cdot)$ is applied element-wise to the values of the diagonal matrix ${\bf S}$, and is defined as
\begin{equation} \label{eq:robust}
f(x) = \sqrt{\bigg(\frac{1-2\alpha}{2\alpha}\bigg)^2 + \frac{x}{\alpha}} - \frac{1 - \alpha}{2\alpha}\;,
\end{equation}
with parameter $\alpha$ set to $0.75$ in practice. The resulting estimate $\hat{ \boldsymbol{\Sigma} }$ can then replace $\boldsymbol{\Sigma}$ in Eq.~\ref{eq:cov_est}.

Thanks to the matrix backpropagation framework of~\cite{Ionescu:2015wa}, which handles eigenvalue decomposition, this robust estimate can also be differentiated, and thus incorporated in an end-to-end learning framework.

\prg{Multiple CDUs.}
Our second strategy to handling high-dimensional feature maps, illustrated by Fig.~\ref{fig:fuse}(left), consists of splitting the feature maps into $n$ separate groups of equal sizes. Each group will then act as input to a different CDU, whose covariance matrix will have fewer 0 eigenvalues than a covariance obtained from all the features. For example, with a ResNet, instead of computing a covariance descriptor of size $2048 \times 2048$, we create 4 groups of 512 features, and use them to compute 4 different covariance descriptors, followed by separate O2T and PV layers. In essence, this strategy still makes use of all the features, but does not consider all the possible pairwise covariances. However, since the features are learned, the network can automatically determine which pairwise covariances are important. Note that the robust covariance estimate discussed above can be applied to the covariance matrix of each group.

Ultimately, the information contained in the multiple CDUs needs to be fused into a single image representation. We propose two strategies to do so, illustrated in Fig.~\ref{fig:fuse}(right). The first one consists of combining the CDUs output vectors by an operation such as summing, averaging or concatenation. The second one aims at fusing the multiple branches before vectorization, which can be again achieved by summing or averaging the respective matrices, or concatenating them into a larger block-diagonal matrix. This is then followed by a PV layer.

\section{Experiments}

In this section, we first present results obtained with our basic SO-CNN introduced in Section~\ref{sec:basic} on CIFAR-10. We then turn to evaluating our complete SO-CNN architecture, with the different strategies introduced in Sections~\ref{sec:transition} and~\ref{sec:robust}, on the larger, more challenging MINC dataset.

\subsection{Basic SO-CNNs on CIFAR-10}
\label{sec:cifar_result}
\textbf{CIFAR-10}~\cite{krizhevsky2009learning} is an object recognition dataset containing 50000 training and 10000 testing $ (32 \times 32)$ RGB images depicting 10 classes of objects. In the following experiments, we augmented the data by flipping the training images for all models and baselines. Because of the relatively small scale of this dataset, we can directly apply our basic SO-CNN to it. We therefore make use of this dataset to evaluate different architecture designs within our basic SO-CNN framework. Furthermore, we compare our basic SO-CNN to the corresponding first-order CNN, to the matrix backpropagation model of~\cite{Ionescu:2015wa} (\emph{MatBP}) and to the \emph{SPD-net} of~\cite{huang2017}.

\begin{figure}
\vspace{-0.2cm}
	\includegraphics[width=\columnwidth]{cv_fitnet_composed.eps}
    \caption{{\bf Joint influence of the PV output dimension and the second-order dimension.}
    With no O2T layers, learning is unstable once the PV dimension becomes significantly larger than the covariance dimension (64). With one O2T layer, learning is more stable, particularly when the PV dimension is not significantly larger than the O2T dimension. This suggests one should use a PV dimension similar to that of the last O2T layer.
    }
    \label{fig:wv_result}
    \vspace{-0.4cm}
\end{figure}

\prg{Model Setup.} We use the FitNet-v1 model of~\cite{Romero:2014tg} as our base first-order architecture. FitNet has 3 convolutional blocks, each of which contains 3 convolutional layers, with no dropout.
The filters are of size $(3,3)$ for all layers, and one max-pooling layer is attached after each block. In the first-order model, the last convolutions are followed by one fully-connected (FC) layer of size 500. In our basic SO-CNNs, we replace this layer with a CDU. Since the last convolutional feature map is of dimension 64, the resulting covariance matrix is sufficiently small not to require a robust estimate or multiple CDUs. Both FitNet and our SO-CNN have then a final FC layer to produce a 10-dimensional vector of class probabilities via a softmax activation. Below, we evaluate different architectures of our SO-CNN model, corresponding to varying the output dimensionality of the PV layer, and the number and dimensionalities of the O2T layers. For all models (first- and second-order), all the weights were initialized using the method of~\cite{Glorot:2010uc}. We used stochastic gradient descent with an initial learning rate of 0.01 and reduced by a factor 10 when the validation loss does not decrease for 8 epochs.

\begin{table}[!t]
\vspace{-0.2cm}
 \centering
  \begin{tabular}[c]{l|l|l|l|l}
    \hline
    \textsc{Setting}  & \begin{footnotesize}SO-CNN-2 \end{footnotesize}& \begin{footnotesize}SO-CNN-3 \end{footnotesize}& \begin{footnotesize}SO-CNN-4 \end{footnotesize}& \begin{footnotesize}SO-CNN-5 \end{footnotesize} \\
    \hline
    \textsc{Same}      &    $82.90\%$    & $83.68\%$  & $83.18\%$  & $84.07\%$ \\
    \textsc{$\div 2$}  &    $82.86\%$    & $84.45\%$  & $83.69\%$   & $83.39\%$ \\
    \textsc{$\times 2$}&    $83.35\%$    & $84.77\%$  & $\boldsymbol{85.10\%}$  & $84.04\%$ \\ \hline
  \end{tabular}
  \vspace{0.2cm}
  \vspace{-0.2cm}
  \caption{{\bf Influence of O2T layer number and dimension.} \textsc{Same} indicates that the dimension is the same (64) in all layers, and $\div 2$ or $\times 2$ that the dimension is divided or multiplied by 2 from one layer to the next. The PV-layer has the same dimension as the last O2T-layer. For example, CDU-3 with $\div 2$ corresponds to O2T(200) - O2T(100) - O2T(50) - PV(50).}
  \vspace{-0.2cm}
  \label{tab:valid_o2t}
  \vspace{-0.6cm}
\end{table}

\prg{PV Output Dimension vs Second-order Dimension.}
Intuitively, the output dimensionality of the PV layer should be similar to that of the second-order descriptor, whether the last O2T layer or directly the covariance when no O2T layer is used (e.g., a much smaller dimension would result in information loss). In a first experiment, we therefore evaluate the joint influence of these two dimensionalities. To this end, we make use of either no O2T layer,
or one such layer
We vary the PV output dimensionality from 10 to 200 with a step size of 10, and the dimensionality of the O2T layer, denoted by O2T($m$) for dimension $m \in \{50, 100,150\}$. In Fig.~\ref{fig:wv_result}, we plot the accuracy of the resulting models as a function of the PV dimensionality.  We can observe that a small $m$ should be used in conjunction with a small PV dimension, whereas a large $m$ yields slightly higher accuracy with a high PV dimension. Furthermore, training seems to be less stable if the PV dimension is significantly larger than the second-order one. We can also see that, as expected, adding one O2T layer brings more flexibility to the model, and thus yields higher accuracy.

\prg{Number and Dimensions of O2T Layers.}
As a second experiment, we evaluate the influence of the number and dimensions of O2T layers in our SO-CNN framework. To this end, we vary the number of O2T layers from 2 to 5 (we also tested with 1 but omit it here due to a consistently slightly lower accuracy), denoted by SO-CNN-\{2,3,4,5\}, and follow three strategies regarding their dimensionalities: (i) We keep the dimension constant across the different O2T layers; (ii) We increase the dimensionality from 50 by a factor 2 in successive O2T layers; (iii) We decrease the dimensionality by half to reach a final dimension of 50. In all these settings, following the results of the previous experiment, we set the PV output dimensionality to that of the last O2T layer.
The results of this experiment are provided in Table~\ref{tab:valid_o2t}. They show that (i) adding more O2T layers indeed increases the capacity of the network, but may lead to overfitting if too many such layers are employed; (ii) the most effective strategy to set the dimensionalities of the O2T layers consists of increasing them in successive layers.

\prg{Comparison to the Baselines.}
Following the previous analysis, in Table~\ref{tab:fitnet_result}, we compare the results of our SO-CNN-4 model with increasing O2T layer dimensions and PV output dimension matching that of the last O2T layer with the first-order FitNet CNN, and the MatBP~\cite{Ionescu:2015wa} and SDP-Net~\cite{huang2017} baselines. For the comparison to be fair, for MatBP, we made use of the same FitNet-based architecture as us. For SDP-net, which relies on a covariance matrix as input, we exploited RCDs obtained from the last convolutional layer of the first-order FitNet.  Note that we were unable to train these two baselines from scratch, as opposed to our SO-CNNs, and therefore fine-tuned them from the pre-trained FitNet. The hyper-parameters of SDP-net were set according to the recommendations in~\cite{huang2017}.

\begin{table}[!t]
\vspace{-0.2cm}
 \centering
  \begin{tabular}[c]{l|l|l|l}
    \hline
    \textsc{Classifier} & \textsc{Settings}
    &\textsc{$\#$ Params} & \textsc{Acc} \\
    \hline
    FitNet~\cite{Romero:2014tg} & 500 & 620K & $83.15\%$ \\
    MatBP~\cite{Ionescu:2015wa} & - & 131K & $28.27\% $ \\
    SPD-net~\cite{huang2017} & 70,50,30 & 55K & $76.07\%$ \\
    \hline
   	\hline
    SO-CNN-4 & $\times 2$ & 362K & $\boldsymbol{85.10\%}$ \\ \hline
  \end{tabular}
  \vspace{0.2cm}
  \caption{{\bf Baseline comparison on CIFAR10 with FitNet architectures.}
 Note that we outperform all baselines, while relying on roughly 40\% fewer parameters than the first-order CNN, which is closest to us in accuracy.}
  \vspace{-0.2cm}
  \label{tab:fitnet_result}
  \vspace{-0.2cm}
\end{table}

As can be seen from the table, our model outperforms MatBP and SPD-net by a significant margin, thus showing the benefits of our end-to-end learning strategy over using a single covariance flattened after a log-map (MatBP) and over a two-stage strategy consisting of using a pre-defined covariance matrix as input (SDP-net). Note also that our model outperforms the first-order one, thus showing the importance of leveraging second-order information. As can be verified from the results of our previous experiments, other versions of our SO-CNN also outperform the first-order one, confirming the benefits of our approach. Altogether, we believe that these results clearly demonstrate the potential of our basic SO-CNN architecture.

\subsection{Complete SO-CNNs on MINC}
We now evaluate our compete SO-CNNs, including the strategies introduced in Sections~\ref{sec:transition} and~\ref{sec:robust}, on the large-scale MINC material recognition dataset. This choice was motivated by the fact that traditional second-order descriptors have proven particularly effective for tasks such as material or texture recognition~\cite{Harandi:2014hy, Harandi:2014hya, Jayasumana:2013we}. Below, we briefly describe this dataset and the architectures we used, as well as evaluate different versions of our approach and compare it to the state-of-the-art.

{\bf MINC-2500} is a large-scale material recognition dataset containing 23 classes of different materials, some of which are shown in Fig.~\ref{fig:minc}. For each class, there are 2500 $(362 \times 362)$ RGB images. We split the dataset into training, validation and test samples with proportions $0.85, 0.05, 0.10$, respectively. Unlike other small-scale material databases~\cite{Caputo:2005gf,Cimpoi:2014vq, Sharan:2009jq}, the images contain not only the material but also its surrounding environment, thus making it more challenging.
To augment the data, we used horizontal flip, and random cropping to $224 \times 224$ patches, thus matching the standard input size of our base CNN architectures described below.

\begin{figure}[t]
\vspace{-0.2cm}
    \centering
    \includegraphics[width=0.9\linewidth]{minc_example.eps}
    \caption{Samples from the MINC-2500 dataset}
    \label{fig:minc}
    \vspace{-0.4cm}
\end{figure}

\prg{CNN Architectures.}
The size of this dataset makes it well-suited to use recent deeper architectures, such as the VGG~\cite{Simonyan:2014ws} and the ResNet~\cite{He:2015tt}. In particular, we use the VGG16 model, which has 16 convolutional layers (configuration C in \cite{Simonyan:2014ws}). For ResNet, we employ the ResNet50 with 50 convolutional layers. To compose our second-order networks, we replace the fully-connected layers and the last average pooling layer with our CDUs. We then attach one fully-connected layer of dimension 23 with softmax activation to obtain the final class probabilities.
For both VGG and ResNet, to reduce over-fitting, we constrain the weights of the O2T layers to be orthonormal. For the comparison to be fair, and following~\cite{Bell:2015ii}, the weights of the common convolutional layers of both first- and second-order models are initialized with weights pre-trained from ImageNet.  In the following experiments, the CDUs all have 3 O2T layers, with  dimensions set to $D$, $\frac{D}{2}$ and $\frac{D}{4}$, where $D$ is the dimension of the covariance matrix. Note that this does not match the best strategy of Section~\ref{sec:cifar_result}, which consisted of doubling the dimension. However, applying this strategy here would result in a final dimension of $2048$,

which would significantly increase computational cost. The PV output dimension is the same as that of the last O2T layer.

\prg{Learning Strategy.}

To train our SO-CNNs (SO-VGG16 and SO-ResNet50), we first freeze the convolutional layers and train the second part of the networks for a few (2-4) epochs, and then fine-tune the whole network in an end-to-end manner. For SO-VGG16, the initial learning rates for second-order training and fine-tuning are set to $10^{-3}$ and $10^{-4}$, respectively, and reduced by a factor 4 when learning plateaued. For SO-ResNet50, the starting rates are set to $10^{-2}$ and $10^{-3}$, respectively.
As mentioned in Section~\ref{sec:transition}, we observed empirically that, during our two-stage learning strategy, we could successfully train the second-order part of the network, but fine-tuning the entire network failed. This, we believe, is due to the fact that, in the first phase, no gradient is backpropagated between the first- and second-order parts of the network. To overcome this, we therefore introduce an additional 1 $\times$ 1 convolutional layer, as described in Section~\ref{sec:transition}. In particular, we set the output dimension of this layer to 512 for SO-VGG16, i.e., the same as the last convolutional layer and to 1024 for SO-ResNet50.
Note that these models still suffer from the low-rank issue discussed in Section~\ref{sec:robust}. Below, we therefore evaluate on our SO-VGG16 model the effectiveness of our different strategies to addressing this issue, introduced in Section~\ref{sec:robust}. We then compare both SO-VGG16 and SO-ResNet50 to the first-order networks and to the same baselines as in the previous section.

\begin{table}[!t]
\centering
\begin{tabular}{|l|l|l|}
\hline
\textsc{Models}         &  \textsc{Fusion} &  \textsc{Accuracy}                       \\  \hline
$2 \times$ CDU          & V-sum       & $67.86\%$                \\ \hline
$2 \times$ CDU          & V-avg       & $75.79\%$                \\ \hline
$2 \times$ CDU          & V-concat    & $75.30\%$               \\ \hline \hline
$4 \times$ CDU          & V-concat    & $74.54\%$  \\ \hline
$8 \times$ CDU          & V-concat    & $76.07\%$  \\ \hline \hline
$2 \times$ CDU          & D-sum       & $75.62\%$                \\ \hline
$2 \times$ CDU          & D-avg       & $76.42\%$                \\ \hline
$2 \times$ CDU          & D-concat    & $\boldsymbol{77.88\%}$               \\ \hline \hline
$1 \times$ CDU + Robust & -           & $74.23\%$               \\ \hline
$2 \times$ CDU + Robust & V-concat    & $76.10\%$               \\ \hline
$2 \times$ CDU + Robust & D-concat    & $75.17\%$               \\ \hline
\end{tabular}
\vspace{0.2cm}
\caption{{\bf Comparison of different SO-VGG16 designs.} {\it Robust} indicates the use of a robust covariance estimate, $n \times$ CDU indicates that the convolutional feature maps are split into $n$ groups with one CDU each. V-\texttt{method} indicates that fusion occurs in vector space, while D- stands for descriptor space.
}
\vspace{-0.2cm}
\label{tab:minc_cv}
\end{table}

\prg{Robust Estimation \& Fusion of CDUs.}
In Section~\ref{sec:robust}, we introduced two strategies to handle high-dimensional feature maps within our SO-CNNs: Making use of robust covariance estimates and exploiting multiple CDUs. In the latter case, we also proposed several ways to fuse the multiple CDUs into a single representation, consisting of summing, averaging or concatenating the vectors output by the CDUs, or performing similar operations on the final second-order descriptors. We will denote these different fusion strategies by \{V,D\}-sum, \{V,D\}-avg and \{V,D\}-concat, respectively, for the vector (V) or descriptor (D) cases. We report the results of these different strategies in Table~\ref{tab:minc_cv}. These results show that (i) making use of multiple CDUs is typically more effective than relying on a robust covariance estimate; (ii) using more than 2 CDUs has little impact; (iii) fusing at the level of second-order descriptors (D) is more effective than at the level of vectors, particularly via concatenation.

\begin{table}[!t]
\centering
\begin{tabular}{|l|l|l|}
\hline
\multicolumn{3}{|l|}{\textsc{Results on VGG16-based model}} \\
\hline
\textsc{Settings}         &  \textsc{$\#$ Params} &  \textsc{Accuracy}                       \\  \hline
VGG16~\cite{Simonyan:2014ws} & 237M                  & $72.14\%$               \\ \hline
$1\times 1$ - FCs         & 237.64M               & $70.13\%$               \\ \hline
MatBP~\cite{Ionescu:2015wa}& 20.77M                & $59.06\%$                \\ \hline
SPD-net~\cite{huang2017}& 0.253M                & $43.90\%$                \\ \hline \hline
Our best                  & 15.21M                & $\boldsymbol{77.88\%}$  \\ \hline
\end{tabular}
\vspace{0.2cm}
\caption{{\bf Baseline comparison on MINC2500 for the VGG16-based models.} We outperform all the baselines significantly, and rely on roughly 90\% fewer parameters than the first-order CNN.}
\vspace{-0.2cm}
\vspace{-0.1cm}
\label{tab:minc_vgg}
\end{table}

\begin{table}[!t]
\centering
\begin{tabular}{|l|l|l|}
\hline
\multicolumn{3}{|l|}{\textsc{Results on ResNet50-based model}} \\
\hline
\textsc{Settings}         &  \textsc{$\#$ Params} &  \textsc{Accuracy}      \\  \hline
ResNet50~\cite{He:2015tt}       & 23.63M                & $80.10\%$              \\ \hline
$1\times 1$ - FCs         & 26.17M                & $80.12\%$               \\ \hline
MatBP~\cite{Ionescu:2015wa}& 32.26M                & $55.35\%$                \\ \hline
SPD-net~\cite{huang2017}& 2.97M                 & $74.33\%$                \\ \hline \hline
Our best                  & 26.00M                & $\boldsymbol{80.45\%}$  \\ \hline
\end{tabular}
\vspace{0.2cm}
\caption{ {\bf Baseline comparison on MINC2500 for the ResNet50-based models.} Note that our SO-ResNet50 again outperforms the second-order-based baselines and the first-order one, although by a smaller margin. We believe that investigating residual second-order strategy could be interesting to further improve our results.
}
\vspace{-0.2cm}
\vspace{-0.2cm}
\label{tab:minc_resnet}
\end{table}

\prg{Comparison to the Baselines.}
In Tables~\ref{tab:minc_vgg} and~\ref{tab:minc_resnet}, we compare the results of our best SO-VGG16 and the corresponding SO-ResNet50 to the first-order CNNs and to the MatBP~\cite{Ionescu:2015wa} and SPD-net~\cite{huang2017} baselines. Since the SPD-net and MatBP models do not implement any robust covariance  estimation,  we  reduced  the  dimensionality of the feature maps to 512 using  two 1$\times$1 convolutional layers. Without this strategy, the models failed to converge.
For the comparison with the first-order models to be fair, we also evaluated a version of these models complemented with the same additional 1$\times$ 1 convolutional layer as in our model. The corresponding models are denoted by \emph{ 1 $\times$ 1 - FCs}.
As in the CIFAR-10 case, our end-to-end approach significantly outperforms MatBP and SPD-net, thus showing the benefits of our framework over simpler second-order-based approaches. Our best SO-VGG16 model also outperforms the first-order VGG16 by a significant margin, while relying on much fewer parameters. Note that this is also true for most of the architectures that we have tested in the previous experiment. The fact that the additional 1 $\times$ 1 convolutional layers yields worse accuracy than the original model evidences that the benefit of our method truly comes from the use of second-order information.
For SO-ResNet50, we used the same strategy as for our best SO-VGG16 model. The comparison between our SO-ResNet50 and the first-order ResNet50 model also turns to our advantage. While the margin here is smaller, we believe that many extensions of our model could be studied to make it even more powerful, such as the notion of residual covariances. This, however, will be a topic for future research.

\section{Conclusion}
In this paper, we have introduced an end-to-end learning framework that integrates second-order information for image recognition. To this end, we have developed new layer types and addressed the practical difficulties arising when dealing with covariance matrices. Our experiments have demonstrated that our framework can outperform first-order networks and other second-order-based baselines. In the future, we will explore alternative learning strategies for this type of architectures. We hope that our research will inspire others to investigate architectures that go beyond the standard first-order ones.

{\small

redpajama_set_name: RedPajamaArXiv


text:
And the People Shouted Hallelujah
jh staff Faith May 24, 2020
In a bold move last week, President Donald Trump announced that his administration would seek to immediately reopen houses of worship across the country.
Next came an order to the Centers for Disease Control and Prevention to classify churches, synagogues, and mosques as "essential places that provide essential services."
"Some governors have deemed liquor stores and abortion clinics as essential, but have left out churches and other houses of worship," President Trump said in the White House press room, punctuating his statement with the words, "It's not right."
"The people are demanding to go to their church and synagogue, to go to their mosque," the president said, adding that in America "we need more prayer, not less."
Spirituality, by virtue of its existence, is essential. In America, its manifestation has historically been safeguarded by the words contained in our inspired foundational document.
Hard to believe that we could ever have been denied the necessity of the soul.
President Trump had another message for officials who have little sense of urgency and seem content to delay indefinitely when it comes to allowing houses of worship to reopen.
"If they don't do it, I will override the governors," the president said.
His remarks have been mischaracterized by the Democrats and the antagonistic media from the moment they were uttered. Many of the same partisan organizations and individuals show little or no regard for a paramount constitutional right—the free exercise of religion.
Some of the so-called experts have weighed in, indicating that President Trump does not have the authority to override governors who are dragging their feet on the reopening of religious institutions.
As head of the executive branch, the president maintains the authority to utilize the Department of Justice (DOJ) to accomplish the objective of securing the cooperation of the governors.
Among the many options, lawsuits can be filed and judges can impose limitations on the actions of governors who are in violation of federal and/or state constitutions. Attorney General William Barr has already demonstrated that he is willing to enter the fray of legal challenges to governors' orders.
The free exercise of religion is included in our First Amendment precisely because the founders understood the essential nature of religious liberty. To ever have given houses of worship a "non-essential" label not only runs counter to the First Amendment, but it has the potential to hinder a primary life process in which an individual and/or groups engage, particularly in times of distress or anxiety.
Our country's first president would have been on board with our current president in understanding the necessity for spirituality and religious expression.
As shown in Arnold Friberg's famous painting "The Prayer at Valley Forge," the image of then-General George Washington on his knees has inspired Americans since the work of art was first unveiled in 1976, the year of our nation's bicentennial.
As the story goes, a young Pennsylvania senator named Isaac Potts was against the war that gave birth to America. His opposition would not last long, though.
One day he happened upon a man who was immersed in deep prayer. At his side a sword lay placid on the ground. The solitary figure turned out to be General Washington himself, asking the Almighty to assist him in his cause of emancipating a nascent country.
Reflecting on the prayer, Potts became convinced that the American Revolution "was the cause of God, and America could prevail."
President George Washington would later say, "Of all the dispositions and habits which lead to political prosperity, religion and morality are indispensable supports."
Perhaps much like something President Washington would have said if faced with the same circumstances, President Trump let governors and officials across the land know that religious institutions, and the worship services they provide, play an essential role.
He has spoken for the searchers whose life-sustaining spirituality is, and always will be, essential.
And the people shouted hallelujah.
Previous Post Woodstock: From Liberty to Lockdown
Next Post What You Need to Know about the Heads of Social Media and Big Tech

redpajama_set_name: RedPajamaCommonCrawl


text:
package com.gensolution.activity.service;

import java.sql.SQLException;
import java.util.List;

import org.mybatis.spring.support.SqlSessionDaoSupport;

import com.gensolution.activity.dao.ActivityCvsDAO;
import com.gensolution.activity.vo.ActivityCvsItemVo;
import com.gensolution.activity.vo.ActivityCvsVo;
import com.vertexid.vo.NaviVo;

public class ActivityCvsService extends SqlSessionDaoSupport implements ActivityCvsDAO {

	@Override
	public NaviVo cvsListCnt(NaviVo naviVo) throws SQLException {
		naviVo.setTotRow((Integer)getSqlSession().selectOne("cvsListCnt" ,naviVo));
		return naviVo;
	}

	@Override
	public List<ActivityCvsVo> cvsList(NaviVo naviVo) throws SQLException {
		return getSqlSession().selectList("cvsList", naviVo);
	}
	@Override
	public List<ActivityCvsItemVo> cvsItemList(String drcInnb) throws SQLException {
		return getSqlSession().selectList("cvsItemList", drcInnb);
	}

	@Override
	public String cvsGetMatter(String drcInnb) throws SQLException {
		return  getSqlSession().selectOne("rndCvsMatter", drcInnb );
	}

}

redpajama_set_name: RedPajamaGithub


Pancreatic tumors are surrounded by a protective "nest" made of fibrotic scar tissue and the cells that manufacture it (red). A new study demonstrates that disrupting this fibrous tissue makes immune therapy and chemotherapy more effective in attacking tumors of the pancreas.
While immune therapy has proven effective in treating certain types of cancer, especially lung cancer and melanoma, tumors of the pancreas remain among the most difficult to treat and, so far, are impervious to immune-based therapies. Now, a new study in mice has shown that immunotherapy against pancreatic cancer can be effective when given in conjunction with drugs that break up the fibrous tissue in these tumors.
The study, from Washington University School of Medicine in St. Louis, appears July 4 in Nature Medicine.
Based on the study's findings, doctors at Siteman Cancer Center at Washington University School of Medicine and Barnes-Jewish Hospital are conducting a phase 1 clinical trial in patients with advanced pancreatic cancer that will test the safety of this drug combination when given along with standard chemotherapy.
Unlike other cancer types, pancreatic tumors are characterized by a large amount of what DeNardo describes as scar tissue. This extra connective tissue and the cells that deposit it provide a protective environment for cancer cells, stopping the immune system from attacking the tumor cells and limiting the cancer's exposure to chemotherapy delivered through the bloodstream. DeNardo and his colleagues investigated whether some of this protection might be lost if they could disrupt the proteins that help fibrous tissue adhere to itself and surrounding cells.
Focal adhesion kinase (FAK) inhibitors have been developed in other areas of cancer research, but DeNardo and his colleagues, including oncologist Andrea Wang-Gillam, MD, PhD, an associate professor of medicine, are the first to test them against pancreatic cancer in conjunction with immunotherapy. In the mouse study, an investigational FAK inhibitor was given in combination with a clinically approved immune therapy that activates the body's T cells and makes tumor cells more vulnerable to attack.
Mice with a model of pancreatic cancer survived no longer than two months when given either a FAK inhibitor or immune therapy alone. Adding FAK inhibitors to standard chemotherapy improved tumor response over chemotherapy alone. But the three-drug combination — FAK inhibitors, immune therapy and chemotherapy — showed the best outcomes in laboratory studies, more than tripling survival times in some mice. Some were still alive without evidence of progressing disease at six months and beyond.
The success of this mouse study provided a strong rationale for testing this drug combination in patients with advanced pancreatic cancer, according to Wang-Gillam.
This work was supported by the Lustgarten Foundation; an AACR/PANCAN Award; the National Cancer Institute (NCI) of the National Institutes of Health (NIH), grant numbers R01-CA177670, R01-CA203890, R21-CA182701; the BJCIH/Siteman Cancer Center Cancer Frontier Fund; and the Washington University Institute of Clinical and Translational Sciences, grant number KL2TR000450.
Jiang H, Hegde S, Knolhoff BL, Zhu Y, Herndon JM, Meyer MA, Nywening TM, Hawkins WG, Shapiro IM, Weaver DT, Pachter JA, Wang-Gillam A, DeNardo DG. Targeting focal adhesion kinase renders pancreatic cancers responsive to checkpoint immunotherapy. Nature Medicine. July 4, 2016.


Federation of Nepalese Chamber of Commerce and Industry (FNCCI) is organising the sixth edition of Nepal International Trade Fair-2017 from March 16.
The exhibition will have 340 stalls showcasing various domestically-made as well as imported products.
According to the organiser, Chinese companies will showcase their products from 85 stalls, while those from India have 10 stalls. Pakistani traders have booked 20 stalls, whereas 50 stalls have been allocated for Bangladeshi firms and two for Thai companies.
Importers of Nepali products from countries like Japan, Korea and Thailand, among others, will also be participating in the event.
FNCCI said the fair will also offer a platform for enhancing B2B contacts and promoting global products under a single roof. Besides, the exhibition aims at boosting technology transfer and identifying potential investment areas. Handicrafts, tourism, information technology, agricultural products, energy, construction materials and interior design products, among others, will be major attractions of the event. Other products on display include drugs and medicines, iron and steel, fashion wear, leather goods and food items.
FNCCI is organising the trade fair in association with International Labour Organisation (ILO) and a number of private sector organisations such as Mercantile Communication, Official Cargo Ganesh, Ganesh Air Cargo, Carpet Sponsor Heritage Carpet, Premier Insurance and Arghakhachi Cement.
The event will also host 35 stalls showcasing herbal products of various domestic companies. FNCCI's Agro Enterprise Centre of FNCCI will showcase agricultural products from 70 stalls. On the sidelines, ILO will be conducting a job fair.
FNCCI has expected more than 200,000 visitors.


The history of 19th century England abounds with great religious figures and, at the same time, religious turmoil. Among those who participated in the Evangelical revival now known as the Oxford Movement were Henry Manning and his three brothers-in-law, Samuel, Robert and Henry Wilberforce, sons of the leading social reformer William Wilberforce. Also within this circle of devout Evangelicals was the influential John Henry Newman, who like Manning and both Robert and Henry Wilberforce, eventually left the Anglican communion for the Roman Catholic Church.


Uncork 2016 with us at PARISH!
Enjoy dinner with us and stay for our Uncorked celebration!
We're 'uncorking' nothing but bubbles to ring in the New Year, so bring your friends and enjoy this special occasion of merriment!
Our a la carte menu will only be available until 9 pm. Our featured Uncorked will pop at 10pm; cost per person is $50, $30 if you enjoy dinner beforehand and stay for Uncorked.
Additional festivities will include live DJ, hats and noisemakers!


text:
Mikel Balenziaga Oruesagasti (Zumarraga, 29 februari 1989) is een Spaans voetballer die doorgaans als linksback speelt. Hij verruilde Real Valladolid in juli 2013 voor Athletic Bilbao.

Clubcarrière
Balenziaga speelde twee seizoenen voor het tweede elftal van Real Sociedad. In 2008 stapte hij over naar rivaal Athletic Bilbao. Op 14 september 2008 debuteerde hij samen met Ander Iturraspe in de Primera División voor Athletic Bilbao tegen Málaga CF. Tijdens het seizoen 2009/10 speelde hij op uitleenbasis voor CD Numancia. In 2011 verhuisde hij naar Real Valladolid, waar hij twee seizoenen verbleef. In 2013 werd hij voor een half miljoen euro teruggekocht door Athletic Bilbao.

Interlandcarrière
Balenziaga haalde vijf caps voor Spanje –19. In 2008 speelde hij zijn enige interland voor Spanje –21.

Erelijst

Spaans voetballer

redpajama_set_name: RedPajamaWikipedia


text:
Scholarships in Switzerland Without IELTS
Scholarships in Switzerland For 2022 Without IELTS
Wide range of courses. Get an internationally recognized degree. Scholarships in Switzerland for 2022 without IELTS for international students. Switzerland is one of the most important countries. The scholarships are available for Bachelor, master, and PhD degree Programs. There are many fully funded scholarships and internship opportunities in Switzerland for Bachelor, master, and Ph.D. students. … Read more
Categories List of Scholarships in Switzerland, List of Switzerland Scholarships Without IELTS, List of Universities in Switzerland Without IELTS, Scholarships, Scholarships in Switzerland For 2022, Scholarships in Switzerland For 2022 Without IELTS, Scholarships in Switzerland For 2022 Without IELTS 2022, Scholarships in Switzerland Without IELTS Leave a comment
Scholarships in Switzerland Without IELTS 2021
Switzerland is one of the most important countries. Scholarships in Switzerland without IELTS for the academic year 2021-22. There are many fully funded scholarships and internship opportunities in Switzerland for Bachelor, master, and PhD students. Some universities in Switzerland where you can be exempted from the IELTS test. Switzerland is also an ideal choice for … Read more
Categories internship opportunities in switzerland, internships in switzerland, Scholarships, Scholarships in Switzerland Without IELTS, Scholarships in Switzerland Without IELTS 2021, Scholarships in Switzerland Without IELTS 2022 Leave a comment
Government of Italy Scholarships 2022


Women La Prairie Anti- Aging Foundation SPF 15 - # 700 Foundation 1 oz 1 pcs sku# 1787289MA. New La Prairie Switzerland Anti- Aging Foundation Sunscreen - Shade 700 See more like this.
These sales have gotten sweeter. LOT OF 2 LA PRAIRIE ANTI- AGING EMULSION 0. The Anti- Aging Collection reawakens skin through unique La Prairie targeted treatments that protect against the effects of time.
Sie möchten günstig Online Parfüm kaufen? 7 results for " la prairie anti aging foundation" Showing selected results.
A financial advisor do you take clomid all at once Karas was taken into custody on Tuesday at his Las Vegas home, 62, best known for reputedly building a beginning stake of $ 50 into a $ 40 million fortune during a record three- year winning streak the San Diego County District Attorney' s Office said on Friday. How- to- Use: Shake well before using. La prairie anti aging foundation preis.
The brand traces its origins to 1930s Switzerland, where a doctor at Clinique La Prairie pioneered cellular anti- aging therapies. More options available: $ 215. Star of numerous movies in his 46- year career; he' s best known for his roles in the movies Apocalypse Now The Execution of Private Slovak , the television series West Wing. Smooth over your moisturized face, blending well from forehead to jaw line.
Anti- Aging Swiss Ice Crystal Skincare Moisturizers Serums exfoliators Specialists UV protection Body , toners Masks , essences Eye , lip care Cleansers hand care Our most coveted Makeup Face Eyes Lips Our most coveted The Art of Gifting Limited Editions Exclusive Offer Our story The essence of La Prairie : la prairie anti aging. We have now placed Twitpic in an archived tdeco Kosmetik folgt stets den aktuellen Trends und ihre Produkte genügen den höchsten Ansprüchen. La prairie anti aging foundation preis.
See all results for la prairie anti aging foundation. Women La Prairie Anti- Aging Foundation SPF 15 - # 700 Foundation 1 oz 1 pcs. La prairie anti aging foundation preis.
Es war Ende der 1970er Jahre als der Ästhetikmediziner Michel Tordjman zusammen mit anderen Medizinern und Schönheitsspezialisten damit begann ein Forschungsprogramm zum Thema Hautalterung durchzuführen. More Buying Choices $ 215. Find great deals on eBay for la prairie anti aging. At finden Sie diverse hochwertige Augen Make- ups für anspruchsvolle Kundinnen.
In unserem Sortiment sind langanhaltende Eyeliner und Flüssig- Eyeliner Profi- Eyeliner verschiedenste Lidschatten- Paletten und selbstverständlich auch gute Mascaras. : la prairie anti aging. Als Nichte eines Apothekers konnte Estée schon sehr früh die Geheimnisse der Kosmetikherstellung kennenlernen. Jahrhundert eine Revolution in der Beauty- Welt in Gang setzte.
For the most effective anti- aging ritual, start with Anti- Aging Rapid Response booster. 60 ( 1 new offer) La Prairie Skin Caviar Luxe Eye Lift Cream, 0.
Check out our Spring deals on la prairie anti- aging foundation spf 15 600, 1 oz at its new low price. Masterfully merging colour flawless complexion, care, it will not only produce a uniform, skilfully concealing imperfections in texture , but will also nourish, firm , tone hydrate your skin while providing excellent UV protection. Shop with confidence. Discover more here.
Skip to main content. Get the perfect gift before it' s gone! In short this luxuriously lightweight yet full- coverage foundation truly fulfils its anti- ageing Prairie skin care, art , fragrance represent an exquisite fusion of Swiss science, makeup rare ingredients. Cialis prescription men Whether in grim housing projects surrounding the cities of.
Unser Produktportfolio enthält Produkte vieler namhafter Marken: Bourjois Lancôme, Max Factor Artdeco u. Apply with a sponge if more coverage is desired. Probieren auch Sie die Lidschatten oder die Make- up- Basen von Artdeco! Martin Sheen – 12- time nominated 3- time Emmy Award winning Golden Globe winning Actor.
Father of actors: Emilio Estévez Ramón Luis Estévez, Charlie Sheen Renée Estévez. Anti- Aging Foundation SPF 15 creates a flawless complexion hydrating , while nourishing, tone, concealing imperfections in texture helps prevent damage from UV rays.
Shop Online La Prairie Anti- Aging Foundation Spf15 200 at best price. Willkommen bei Perfumetrader Ihrem Online Shop für Parfüm Kosmetik Make- Up und Pflegeprodukte. Buy La Prairie Foundations - La Prairie Anti- Aging Foundation SPF 15 200. This high- performance anti- aging makeup is the perfect combination of color and skincare.
la prairie ist ein Kosmetikspezialist par excellence mit Hauptsitz in Montreux in der Schweiz. Das Traditionsunternehmen arbeitet stets an Innovationen, die die Haut auf Zellebene Prairie – Die Kunst der Schönheit Aus der renommierten Schweizer Clinique La Prairie in Montreux ist La Prairie hervorgegangen.
Seit 1982 gibt es La Prairie als eigenständiges Unternehmen, dessen Fokus auf Pflegeprodukten gegen den Hautalterungsprozess liegt. Filorga ist eine französische Kosmetik- Marke, die sich auf die Entwicklung hochwirksamer Anti- Aging- Lösungen spezialisiert hat.