text:
\section{Graph Attention Networks}
\label{sec:app:models}

The original paper \textit{Velickovic et al.} \cite{Velickovic:2018we} proposed a new approach to graph representation learning incorporating the popular attention mechanisms of sequence-based tasks.

The basic concept is to provide and attention mechanism $\bm{a}$ which calculates the attention coefficients between a node $v_i$ and its neighbours ${\mathbb{N}}_i$, employing self-attention. This attention coefficient between node $v_i$ and a neighbour $v_j$. These coefficients are then normalised to compare across nodes using the \textit{softmax} transform resulting in the final coefficient for a given neighbour $\alpha_{ij}$

\begin{equation}
    \label{eq:GAT}
    \alpha_{ij} = \frac{exp(\mbox{LeakyReLU}(\bm{a}^T[\bm{\theta}\bm{h}_i||\bm{\theta}\bm{h}_j]))}{\sum_{k \in {\mathbb{N}}_i} exp(\mbox{LeakyReLU}(\bm{a}^T[\bm{\theta}\bm{h}_i||\bm{\theta}\bm{h}_j]))}
\end{equation}

This attention mechanism occurs for each head in a layer allowing for multi-headed attention. The attention coefficient and weight matrices for each head are denoted with a superscript $k$ where $k \in [K]$ and $K$ is the total number of heads. We can thus relate GAT to the basic MESSAGE-PASSING network equation \ref{eq:messagepassing} as such

\begin{equation}
    \bm{h}^l_i = \mbox{AGGREGATE}_k^K (\sigma(\sum_{j \in {\mathbb{N}}_i} \alpha^k_{ij}\bm{\theta}^k \bm{h}^{l-1}_j))
\end{equation}

where AGGREGATE is some aggregation function, the original paper used MEAN and CONCATENATE.

A later paper \textit{Brody et al.} \cite{Brody2021GATv2} questioned the attentiveness of the proposed method and provided the following attention mechanism instead of equation \ref{eq:GAT}

\begin{equation}
    \alpha_{ij} = \frac{exp(\bm{a}^T\mbox{LeakyReLU}(\bm{\theta}[\bm{h}_i||\bm{h}_j]))}{\sum_{k \in {\mathbb{N}}_i} exp(\bm{a}^T\mbox{LeakyReLU}(\bm{\theta}[\bm{h}_i||\bm{h}_j]))}
\end{equation}

This graph connection approach allows us to train the attention mechanism $\bm{a}$ across the graph $\mathcal{G}$ without having to train the weight matrix $\bm{\theta}$. So if we are provided with suitable weights we can greatly reduce the training cost.
\section{Background}
\label{sec:background}
\input{tables/dataset-overview}

We compare our new method (GraNet) against some standard Graph Neural Networks to demonstrate the improvements that GraNet makes in classifying datasets.

\citet{Kipf:2017tc} introduce GCN (Graph Convolutional Networks) -- a method of applying convolutional layers from CNNs to graph neural networks. It focuses on spectral filters applied to the whole graph structure rather than at the node level.

\citet{Hamilton:2017tp} introduce the GraphSAGE model which builds on prior work from GCN focusing on individual node representations. This gives rise to the iterative message passing process on the node level. Though simpler than newer models we find that this approach, when given the right embedding style, can outperform some recently published GNNs.

\citet{Velickovic:2018we} introduce the idea of graph attention which alters how a node aggregates its neighbours representation. This adds an additional attention mechanism to discern which aspects of the node representations in a nodes neighbourhood are important at a given layer.

\citet{Brody2021GATv2} provide a more attentive version based on the graph attention system introduced in \citet{Velickovic:2018we}. We base the graph attention mechanism used in our GraNet models on this improved version of graph attention. We provide both versions of graph attention in our results to compare to our new approach.

\subsection{Notations}
\textbf{Graph Data}
\label{sec:background:preliminary}
Let ${\mathcal{G}}({\mathbb{V}}, \mathcal{E}, {\bm{X}})$ denote a graph where ${\mathbb{V}} = \{{\bm{v}}_1, {\bm{v}}_2, ..., {\bm{v}}_n\}$ is the set of nodes and $N=|{\mathbb{V}}|$ is the number of nodes in the graph, $\mathcal{E}$ is the set of edges between nodes in ${\mathbb{V}}$ such that ${\bm{e}}_{i,j} \in \mathcal{E}$ denotes a directed connection from node ${\bm{v}}_i$ to node ${\bm{v}}_j$, ${\bm{e}}_{i,j}$ may itself be a feature vector.
We say each node ${\bm{v}}_i$ has a neighbourhood ${\mathbb{N}}_i$ such that ${\bm{v}}_j \in {\mathbb{N}}_i \iff {\bm{e}}_{j,i} \in \mathcal{E}$ and we say that ${\bm{v}}_j$ is a neighbour node to ${\bm{v}}_i$. Where ${\bm{X}}$ is the raw data matrix where ${\bm{X}}_{:, i} = {\bm{x}}_i$ where ${\bm{x}}_i$ is the feature vector for node ${\bm{v}}_i$.

\textbf{Embeddings}
\label{eq:embedding}
There normally exists a transformation function, $f_e$, to project the raw data to a more compact feature ${\bm{X}}_e$ space such that ${\bm{X}}_e = f_e({\bm{X}})$

For instance, we can transform a set of images (${\bm{X}} \in {\mathbb{R}}^{N \times C \times H \times W}$, where $C$, $H$ and $W$ are the number of channels, width and height of an image) to 1D features (${\bm{X}}_e \in {\mathbb{R}}^{N \times F}$, where $F$ denotes the feature dimension). In this case, we have an embedding function $f_e: {\mathbb{R}}^{N \times C \times H \times W} \rightarrow \mathbb{R}^{N \times F}$ for the dimensional reduction.

This paper puts a special focus on the design of $f_e$, and reveals later how the design choice of $f_e$ can influence the performance of GNN models without making any changes to the underlying data ${\mathcal{G}}({\mathbb{V}}, \mathcal{E}, {\bm{X}})$. An overview of popular datasets, and the embeddings that they use, is presented in \Cref{tab:dataset-overview}. We see that the popular graph datasets \citep{zeng2019graphsaint,Kipf:2017tc,shchur2018pitfalls,Hamilton:2017tp} focus heavily on Bag of Words (BoW) and word vectors. This implies that current GNNs are being tested on and designed for a very narrow class of embedding styles.
A more detailed discussion is available in \Cref{sec:app:dataset-overview}.

\textbf{Graph Neural Networks} Current GNNs can be thought of as \emph{message passing} layers, the $l$-th layer can be represented as

\begin{equation}
\label{eq:messagepassing}
{\bm{h}}_i^{l} = \gamma_{{\bm{\theta}}_\gamma} \left({\bm{h}}^{l-1}_i,
        \psi_{j \in {\mathbb{N}}(i)} \left( \phi_{{\bm{\theta}}_\phi}
        \left({\bm{h}}^{l-1}_i, {\bm{h}}^{l-1}_j,{\bm{e}}_{j,i}\right) \right) \right)
\end{equation}

where $\psi$ is a differentiable aggregation function and $\gamma_{{\bm{\theta}}_\gamma}$ and $\phi_{{\bm{\theta}}_\phi}$ represent differentiable functions with trainable parameters ${\bm{\theta}}_\gamma$ and ${{\bm{\theta}}_\phi}$ respectively. ${\bm{h}}^l_i$ is the node representation of $v_i$ at layer $l$, with ${\bm{h}}^0_i = f_e({\bm{x}}_i)$.

We focus on the improved graph attention mechanism \citep{Brody2021GATv2} when looking at potential GNNs for GraNet. Using this new notation we can formulate it as such

\begin{equation}
    \label{eq:GATv2}
    \alpha_{ij} = \frac{exp(\bm{a}^T\mbox{LeakyReLU}(\bm{\theta}[\bm{h}_i||\bm{h}_j]))}{\sum_{k \in {\mathbb{N}}_i} exp(\bm{a}^T\mbox{LeakyReLU}(\bm{\theta}[\bm{h}_i||\bm{h}_j]))}
\end{equation}

where $\bm{a}$ is a learnable parameter representing the attention of the network. Further discussion of graph attention and why this is useful in GraNet is available in \Cref{sec:app:models}.

\textbf{Pre-trained models} are specific neural network architecturwa that have been trained on a dataset $\mathcal{D}$ for a specific task, this could be ImageNet classification \citep{deng2009imagenet} for vision networks \citep{he2016deep} or the entire English Wikipedia for language models \citep{liu2019roberta}. These networks therefore have \textit{pre-trained} weights $\bm{\theta}$ that can be loaded into the model for further training or evaluation.

We denote these pre-trained models as $f_{{\bm{\theta}}}$ that is parameterised by weights $\bm{\theta}$. We say that a pre-trained model has a set of functions $\{f^1_{{\bm{\theta}}_1}, f^2_{{\bm{\theta}}_2}, ..., f^M_{{\bm{\theta}}_M} \}$ for an $M$-layer model, where $f^i_{{\bm{\theta}}_i} : {\mathbb{R}}^F \rightarrow {\mathbb{R}}^{F^\prime}$ and $f^{i+1}_{{\bm{\theta}}_{i+1}} : {\mathbb{R}}^{F^\prime} \rightarrow {\mathbb{R}}^{F^{\prime\prime}}$ and $F$ is the feature dimension.
A pass through a single layer, $l$, of a network would be $f^l_{{\bm{\theta}}_l}({\bm{x}})$, shorthand for $f^l({\bm{x}}; {\bm{\theta}}_l)$.
If we concatenate these layers to form a full pass through the network, we obviously have $f_{{\bm{\theta}}}({\bm{x}}) = f^M_{{\bm{\theta}}_M}(...f^2_{{\bm{\theta}}_2}(f^1_{{\bm{\theta}}_1}({\bm{x}})))$

\textbf{Fine-tuning} is therefore adapting ${\bm{\theta}}$ to a new dataset $\mathcal{D}^\prime$ which is related to $\mathcal{G}({\mathbb{V}}, \mathcal{E}, {\bm{X}})$ as illustrated in \Cref{sec:background:preliminary}.
In a more standard setup, the target dataset we fine-tune to has the same underlying data-structure as the pre-training dataset, for instance, they might both be images, but the target dataset is a different type of classification. This may involve adding, removing or altering specific layers within the model or simply retraining the model with different labels on $\mathcal{D}$.

If the architecture of a pre-trained model is altered then a new weight matrix ${\bm{\theta}}^\prime$ must be created from ${\bm{\theta}}$ by adding, removing or reshaping weights.

\textbf{Freezing} layers is the process whereby a selection of weights ${\bm{\theta}}_f \subset {\bm{\theta}}$ do not have gradients and thus do not change during back-propagation.

These ideas allow us to alter these pre-trained models to use information about the graph connections whilst utilising their pre-trained weight ${\bm{\theta}}$.

\textbf{Blending and Fine-tuning} models is therefore the process of using existing models, $f_{{\bm{\theta}}_f}$ and $g_{{\bm{\theta}}_g}$, with defined set of layers, $\{f^1_{{\bm{\theta}}_{f1}}, f^2_{{\bm{\theta}}_{f2}}, ..., f^N_{{\bm{\theta}}_{fN}} \}$ and $\{g^1_{{\bm{\theta}}_{g1}}, g^2_{{\bm{\theta}}_{g2}}, ..., g^M_{{\bm{\theta}}_{gM}} \}$, and creating a new model, $h_{{\bm{\theta}}_{f,g}}$, such that $h^i_{{\bm{\theta}}_{f,g}} = f^j_{{\bm{\theta}}_f} \circ g^k_{{\bm{\theta}}_g}$. We can use pre-trained weights ${\bm{\theta}}_f$ or ${\bm{\theta}}_g$ and/or freeze either model, and where one of these models is a GNN we say this is fine-tuning on a graph dataset.

\textbf{Unconnected models} are models that, unlike GNNs, do not use any information about graph connections within a dataset. These are trained on datasets that are not graph-connected where each datapoint is consider isolated. We focus on complex unconnected models with multiple layers, such as vision networks, which we call \textit{large models}. Due to training cost we use \textit{pre-trained large models}.
\section{Conclusion}
In this paper, we reveal that GNN model designs are overfitting to certain embeddings styles (\textit{e.g.} BoW and word vectors). To demonstrate this we introduced three new datasets each with a range of embedding styles to be used as a more comprehensive benchmark of GNN performance.

We demonstrated that embedding style influences the performance of GNNs regardless of the underlying dataset. Equally, the quality of the embedding, measured by how well an unconnected baseline model performs, is a greater indicator of GNN accuracy than the GNN architecture chosen. \emph{We therefore stress the importance of creating high quality embeddings} and \emph{choosing the best GNN architecture based on the style of embedding} created rather than using (or trying to improve) the same GNN model for every task.

We then introduced a new approach named GraNet. \emph{This approach allows for any large pre-trained model to be fine-tuned to a graph-connected dataset} by altering the standard message passing function. In this way we exploit graph structure information to enhance the pre-trained model performance on graph-connected datasets. We have demonstrated that \emph{GraNet outperforms both unconnected pre-trained models and GNNs} on a range of datasets.

There is an increasing trend towards large pre-trained models and graph-connected datasets.
Our work demonstrates potential pitfalls in the way GNN architectures are currently evaluated and proposes a new technique to fully exploit the benefits of pre-trained models within a GNN.
\newpage
\section{Our Datasets}
\label{sec:app:datasets}
\input{tables/dataset-flickr}

In this paper we look at the Flickr \citep{Hamilton:2017tp} and Amazon \citep{Hamilton:2017tp, shchur2018pitfalls} datasets (specifically looking at the Electronics and Instruments subsection of Amazon). An overview of the provided datasets and embeddings is present in \Cref{tab:flickr-dataset}.

As demonstrated in \Cref{sec:related} the process of building a graph dataset is currently signified by choosing an embedding function $f_e$ to apply to the underlying dataset $\bm{X}$. We provide three cases of the underlying dataset $\bm{X}$ and provide a selection of embeddings from sensible embedding functions. We also provide a \textit{raw} graph-connected image dataset, by which we mean the underlying image dataset $\bm{X}$ is not reduced to a matrix of 1D feature vectors but remains a multi-dimensional tensor of multi-dimensional image tensors.

As discussed in \Cref{sec:related}, versions of these datasets \citep{zeng2019graphsaint,shchur2018pitfalls}
utilize classical word embeddings. We propose using large pre-trained models as embedding functions as these can extract richer features ($\displaystyle {\bm{X}}_e$) that may better represent the underlying data ($\bm{X}$) of the dataset.

We will host our datasets in a website and make both these baselines and datsets accessible to the community (website link will be added after acceptance).

\subsection{Flickr\_v2}
The prior Flickr \citep{zeng2019graphsaint}
dataset does not contain the raw image data nor the flickr identifiers only the embeddings created in \textit{McAuley et al.} \cite{McAuley2012image} but
SNAP provides an adjacency matrix and flickr identifiers. We follow how \textit{Zeng et al.} \cite{zeng2019graphsaint} generated their dataset by downloading all the images still available, but we are unable to match the exact images they sourced or the labels. This does mean we cannot directly compare results but we find that the benchmark results we have achieved are similar.

As shown in \Cref{tab:dataset-overview} Flickr uses BoW but as discussed in \Cref{sec:setup:flickr} we deem this inappropriate for the image dataset. A potential solution would be to process the images, as discussed, to create text descriptions that can be converted into BoW embeddings. However, this would either require a human, and thus the benefits of using a neural network (such as time efficiency of classification) is lost, or a neural network. In the case of a neural network we claim that the provided networks for image classification would perform better. This is an area for further investigation especially as we found BoW works better in the case of Amazon Instruments.

As the dataset is created from four separate image classification challenges \citep{Everingham15, nus-wide-civr09, huiskes08} the existing labels and tags for these datasets are not well aligned. We therefore only select the NUS-Wide \citep{nus-wide-civr09} subsection. To create labels for our dataset we use GloVe to generate vector representations of the NUS-Wide tags and find the closest vector representation of 7 hand-picked labels.

The hand-picked labels: [``People", ``Buildings", ``Places", ``Plants", ``Animals", ``Vehicles", ``Scenery"]

This new dataset we call Flickr\_v2.

\subsection{AmazonElectronics and AmazonInstruments}
We use the review text as our raw input features, this means a number of Amazon items are unsuitable as they do not have any reviews. The connections between nodes are based on Amazons three similarity metric ``Co-viewed", ``Co-bought" and ``Similar Items". We only consider direct connections between nodes in a specific Amazon category (in our case Electronics and instruments) making sure removing any nodes that are not directly connected to another node.

Amazon products do not belong to a single category and therefore a products categories are closer to tags than labels. Thus, similar to Flickr\_v2, we use GloVe word vectors to label our dataset based on the set of categories that each product belongs to against a hand-picked selection of categories.

Hand-picked labels for Electronics: [``Camera Photo and Lighting", ``Audio and Video", ``Bags, Cases and Covers", ``Batteries and Chargers", ``Peripherals, Keyboards and Mice", ``Storage and Networking"]

Hand-picked labels for Instruments: [``Electric Guitar", ``Acoustic Guitar", ``Percussion", ``Live Sound \& Stage", ``Studio Recording Equipment", ``Microphones \& Cable", ``Amplifiers \& Effects"]

These new datasets we call AmazonElectronics and AmazonInstruments.

The statistics for these datasets is available in \Cref{tab:flickr-dataset}
\section{Definitions}
\label{sec:app:definition}

\textbf{Pre-trained models} are models where a specific neural network architecture exists that have been trained on a dataset $\mathcal{D}$ for a specific task, this could be ImageNet classification \citep{deng2009imagenet} for vision networks \cite{he2016deep} or the entire English Wikipedia for language models \cite{liu2019roberta}. These networks therefore have \textit{pre-trained} weights $\bm{\theta}$ that can be loaded into the model for further training or evaluation.

We denote these pre-trained models as $f_{\bm{\theta}}$ that is parameterised by weights $\bm{\theta}$. We say that a pre-trained model has a set of functions $\{f^1_{\bm{\theta}_1}, f^2_{\bm{\theta}_2}, ..., f^M_{\bm{\theta}_M} \}$ for an $M$-layer model, where $f^i_{\bm{\theta}_i} : \mathbb{R}^F \rightarrow \mathbb{R}^{F^\prime}$ and $f^{i+1}_{\bm{\theta}_{i+1}} : \mathbb{R}^{F^\prime} \rightarrow \mathbb{R}^{F^{\prime\prime}}$ and $F$ is the feature dimension.
If we concatenate these layers, we obviously have $f_{\bm{\theta}}(x) = f^M_{\bm{\theta}_M}(...f^2_{\bm{\theta}_2}(f^1_{\bm{\theta}_1}(x)))$

\textbf{Fine-tuning} is therefore adapting $\bm{\theta}$ to a new dataset $\mathcal{D}^\prime$ which is related to $\mathcal{G}(\mathcal{V}, \mathcal{E}, \bm{X})$ as illustrated in \Cref{sec:related:preliminary}.

In a more standard setup, the target dataset we fine-tune to has the same underlying data-structure as the pre-training dataset, for instance, they might both be images, but the target dataset is a different type of classification. This may involve adding, removing or altering specific layers within the model or simply retraining the model with different labels on $\mathcal{D}$.

If the architecture of a pre-trained model is altered then a new weight matrix $\bm{\theta}^\prime$ must be created from $\bm{\theta}$ by adding, removing or reshaping weights.

\textbf{Freezing} layers is the process whereby a selection of weights $\bm{\theta}_f \subset \bm{\theta}$ do not have gradients and thus do not change during back-propagation.

These ideas allow us to alter these pre-trained models to use information about the graph connections whilst utilising their pre-trained weight $\bm{\theta}$.

\textbf{Blending and Fine-tuning} models is therefore the process of using existing models, $f_{\bm{\theta}_f}$ and $g_{\bm{\theta}_g}$, with defined set of layers, $\{f^1_{\bm{\theta}_{f1}}, f^2_{\bm{\theta}_{f2}}, ..., f^N_{\bm{\theta}_{fN}} \}$ and $\{g^1_{\bm{\theta}_{g1}}, g^2_{\bm{\theta}_{g2}}, ..., g^M_{\bm{\theta}_{gM}} \}$, and creating a new model, $h_{\bm{\theta}_{f,g}}$, such that $h^i_{\bm{\theta}_{f,g}} = f^j_{\bm{\theta}_f} \circ g^k_{\bm{\theta}_g}$. We can use pre-trained weights $\bm{\theta}_f$ or $\bm{\theta}_g$ and/or freeze either model, and where one of these models is a GNN we say this is fine-tuning on a graph dataset.
\section{Discussion}

\textbf{Discuss smoothness of the embeddings}
\section{Ethics Statement}

With all computer based research there is the issue of carbon footprint. Each train, validation and test run of a model requires electricity which is currently largely sourced from fossil fuels. This training cost carbon footprint is present in this picture, and attempts to keep this low by running short test on new iterations rather than long runs which could waste GPU time and energy.
\section{Evaluation}
\label{sec:eval}

\subsection{Re-evaluating Embeddings}

\textbf{Language Tasks}
\Cref{tab:amazon} demonstrates that the particular embedding function used determines which GNN model performs best on the dataset.
For instance, GAT2 has the best performance if the data is embedded as BoW (Bag of Words), but performs poorly on other embeddings generated from RoBERTa. GraphSAGE, in contrast, performs poorly on BoW but shows a good performance otherwise. Thus which embeddings are used when comparing models has a large effect on which model appears to be the better model. We see the same effect in \Cref{tab:instruments}. Comparing the two tables we see that in the case of the RoBERTa Encoded and RoBERTa embeddings the order of the models changes, this likely comes from the training effects described in \citet{shchur2018pitfalls}.

\Cref{tab:amazon} also signifies the importance of good embeddings as in this case BoW is better than RoBERTa. The complexity of the embeddings does not necessarily improve the efficiency of the classification. This ties into the results of \citet{nt2019revisiting} as the BoW is more informative of the classification containing the label words in the BoW vector. What is also important to note is that when looking at the performance of the GNN against the unconnected model ($\Delta \uparrow$) we see consistency in the difference across the RoBERTa encodings. \emph{This suggests that the ``quality", denoted by how well the simple MLP performs, is a strong indicator of how well a model will perform, rather than the model architecture itself.}

We see an overall decrease in accuracy in \Cref{tab:instruments} across the models but this attributed to the fact that there are more classes for the dataset. We also see that for RoBERTa embeddings the MLP performs poorly though it does occasionally improve on the simpler models, primarily GCN. With more resources it would be better suited to fine-tune RoBERTa (or some other language transformer) to our dataset and use this model as our unconnected model.

\input{tables/electronics}
\input{tables/instruments}

\textbf{Vision Tasks}
\Cref{tab:flickr} demonstrates the same pattern, that the embedding function (in this case a pre-trained vision model) influences which GNN performs the best. It is interesting to note that we see that none of the GAT models achieve the highest accuracy on any of the Flickr\_v2 embeddings. Instead, similar to the RoBERTa embeddings, we see that GraphSAGE performs the best.

It is important to note for VGG16 that the surprisingly poor performance of GNNs is more likely due to the large vector size of more than 25K. With a smaller embedding space better results on par with ResNet may be achieved. Of course, there is also the possibility that the embeddings provided by VGG16 are inferior to ResNet.

\input{tables/flickr}

\Cref{tab:instruments,tab:amazon}, compared to \Cref{tab:flickr}, have far larger increases in accuracy from the best performing GNN compared to the unconnected models. This is mainly due to the fact that we were unable to fully fine-tune RoBERTa to our datasets given limited hardware and time. We hypothesise that the improvements seen in \Cref{tab:amazon} would be smaller when compared to a fine-tuned RoBERTa. Similarly, we did not attempt to create a GraNet model using VGG16 as the results on Flickr\_v2 are worse than the ResNet models and therefore for Flickr\_v2 a GraNet ResNet is ideal.

\Cref{tab:instruments,tab:amazon}, specifically with BoW, are the only instances where GAT and GAT2 are the best models. Furthermore, the entries for BoW follow results from past papers, in contrast to the results shown on all other embeddings. With the prevalence of BoW as shown in \Cref{tab:dataset-overview} it begs the question as to whether we are optimising for BoW extraction rather than graph information extraction.

From these results in both language and vision tasks, we make the following key observations:
\begin{itemize}[noitemsep,nolistsep]
    \item The function $f_e$ used to extract \emph{embeddings influences the performance of different GNNs}, so the embeddings should influence the choice of GNNs regardless of the underlying data.
    \item GNN models do not always outperform simple unconnected models, \emph{graph structure is not enough} to compete against good classifiers.
    \item The choice of an \emph{embedding function contributes more to the final performance} compared to the choice of a GNN model.
\end{itemize}

\subsection{GraNet}

\Cref{tab:granet} demonstrates that extending our standard models with graph connections provides a significant improvement. We see that these GraNet models beat the best performing GNN. The table includes two forms of GraNet models, the fine-tuned models and the trained models.

\textbf{Fine-tuned}
In Flickr\_v2, both GraNet and Unconnected Model use either ResNet18 or ResNet50 weights pre-trained on ImageNet \citep{deng2009imagenet} and then fine-tuned on the target dataset. We observe a significant increase ($+1.5 \%, +1.8 \%$) from the GraNet style of training compared to both unconnected models, and improvement ($1.1 \%, 1.7 \%$) on the best performing GNN model.

Intuitively, GraNet layers reframe graph representation learning from training a GNN on a fixed pre-extracted embedding to training both the GNN and the embedding function ($f_e$) together on the underlying data. There are obviously trade-offs between time and versatility as rather than just training a GNN the embedding function must be trained as well.
But the current approach of GNN training on images requires human, or other annotators, which is a hidden time cost. By combining the embedding function and GNN we provide a more general model that does not need external annotators and thus can work on any raw images.

\textbf{Trained from Scratch}
In the Amazon datasets we do not have any pre-trained and train the GraNet model from initialised weights rather than fine-tuning. However, we still observe an improvement on both the unconnected model and best performing GNN. The improvement is smaller than in Flickr\_v2 ($+0.1 \%, +0.3 \%$ compared to $1.1 \%, 1.7 \%$). But the training time is quicker ($\sim1$hr compared to $\sim10$hrs for 300 epochs).

In this case we do not have a general model as we rely on BoW embeddings for the GraNet model as well. But we can see that improvements are possible on existing GNN techniques on the embeddings themselves, without having to use large pre-trained models.

\input{tables/granet}

We also make the following key observations:

\begin{itemize}[noitemsep,nolistsep]
    \item Graph-connecting a pre-trained network \emph{improves performance by fine-tuning feature extraction based on the graph structure}.
    \item GraNet models \emph{outperform their counterparts by facilitating fine-tuning} on the graph dataset.
\end{itemize}

The architecture for each GraNet is a mixture of the best performing GNN for that embedding and the embedding extraction model ($f_e$) (a Multi-Layer Perceptron in the case of Bag of Words).

\section{GraNet}
\label{sec:app:setup}

Below are the specifics of the GraNet models we present in this paper

\subsection{Flickr\_v2 GraNet}

Graphly connecting every layer in a large pre-trained model is hardware intensive. Therefore, rather than graph-connect every single layer in a pre-trained network $f_{\bm{\theta}}$ we can graph-connect only the final layer. We thus split the model into two portions: the first set of unconnected layers and the final GraNet layer.

We can therefore look at equation \Cref{eq:granet} and see in this case that we partially carry out the forward pass of $f_{\bm{\theta}}$ which we will denote as $f_e$ and then carry out a forward pass through a GraNet layer. This allows us to ignore all the steps involved in applying $f_e$ and focus on the final GraNet layer. Denoting this final GraNet as $g$ we achieve the following equation

\begin{equation}
    y = g(f_e(\bm{X}))
\end{equation}

where y is our output for the given task.

$f_e(\bm{X})$ is the same as described in \Cref{sec:related} and indeed if we were to freeze $f_{\bm{\theta}}$ this would be equivalent to training GraNet with a single layer on the embeddings created by embedding function $f_e$. So instead we also allow $f_e$ to train thus fine-tuning the weights $\bm{\theta}$ of $f_{\bm{\theta}}$. This indirectly allows $f_{\bm{\theta}}$ to learn the graph structure by providing $g$ with better embeddings.

\subsection{Amazon GraNet}

In the case of the Amazon dataset we found that bag of words embedding performed the best. As this does not have an associated pre-trained model we design a multi-layer perceptron (MLP) as a baseline. We then convert all the layers within the MLP to GraNet layers.

However, a single layer of an MLP is $f^l_{\bm{\theta}_l} \left( \bm{x} \right) = \bm{\theta}_l \bm{x}$. This would therefore mean that \Cref{eq:granet} becomes

\[
\alpha_{ij} = \frac{exp(\bm{a}^T\mbox{LeakyReLU}([\bm{\theta}_l\bm{h}_i||\bm{\theta}_l\bm{h}_j]))}{\sum_{k \in \mathcal{N}_i} exp(\bm{a}^T\mbox{LeakyReLU}([\bm{\theta}_l\bm{h}_i||\bm{\theta}_l\bm{h}_j]))}
\]

This is incredibly similar to \Cref{eq:GATv2,eq:GAT} with the only difference being when we apply the weight matrix, concatenation and attention mechanism. We find this model behaves the same as GAT2 and therefore, given the small size tried a different approach as a comparison.

We introduce a new weight matrix $\mathbf{A}$, to replace $\bm{a}$, as our attention matrix. This allows more complex interactions between the node representations to be exploited by our attention. As demonstrated in \Cref{tab:flickr} in \Cref{sec:app:results} we do see an increase in accuracy.

This approach was too costly to apply to a pre-trained CNN as the size of $\mathbf{A}$ far too large.
\section{Hyperparameters}
\label{sec:app:hyperparameters}

\Cref{tab:hyperparameters} details the layers of each model used providing the output hidden features of each layer, the sampler used (the specifics shown in \Cref{tab:samplers}) and the maximum and minimum learning rates. Where there is a difference in learning rates we use a learning rate scheduler that decreases the learning rate when validation accuracy plateaus. Where two models use the same sampler the parameters of those samplers are identical to keep consistency across the tests.

In the case of GraNet we start with a high learning rate for 20 epochs with the pre-trained model frozen to allow the GNN to train. We then unfreeze the entire model dropping the learning rate sharply and train for another 100 epochs. In the first stage no learning rate scheduler is employed, same as for all GNNs, and in the second stage we apply the learning rate scheduler.

For GraphSAINTSampler all setups use a walk length of 2 with 5 steps sampling 100 nodes per node for normalisation calculation.

\input{tables/hyperparameters}
\input{tables/sampler-setup}
\section{Introduction}

Graph Neural Networks (GNNs) have been successful on a wide array of applications ranging from computational biology \citep{zitnik2017predicting} to social networks \citep{Hamilton:2017tp}. The input for GNNs, although sourced from many different domains, is often data that has been preprocessed to a computationally digestible format.
These digestible formats are commonly known as embeddings.

Currently, improvements made to GNN architecture are tested against these embeddings and the state of the art is determined based on those results. However, this does not necessarily correlate with the GNNs accuracy on the underlying dataset and ignores the influence that the source and style of these embeddings have on the performance of particular GNN architectures. To test existing GNN architectures, and demonstrate the importance of the embeddings used in training them, we provide three new datasets each with a set of embeddings generated using different methods.

We further analyse the benefit of using GNNs on fixed embeddings. We compare GNNs to standard models that have been trained or fine-tuned on the target raw data; these models treat each data point as unconnected, ignoring the underlying graph information in data. This simple unconnected baseline surprisingly outperforms some strong GNN models. This then prompts the question: \emph{Will mixing the two approaches unlock the classification power of existing unconnected models by allowing them to utilize the graph structure in our data?}

Based on the question above, we propose a new method of mixing GNNs with unconnected models, allowing them to train simultaneously. To achieve this we introduce a variation of the standard message passing framework. With this new framework a subset of the unconnected model's layers can each be graph-connected -- exploiting useful graph structure information during the forward pass. We demonstrate that this new approach improves the accuracy of using only a pre-trained or fine-tuned  model and outperforms a stand-alone GNN on a fixed embedding.

We call this new approach \textbf{GraNet} (\underline{Gra}ph-connected \underline{Net}work), and in summary, this paper has the following contributions:

\begin{itemize}[noitemsep]
    \item We provide new datasets and a rich set of accompanying embeddings to better test the performance of GNNs.
    \item We empirically demonstrate that only some existing GNNs improve on unconnected model accuracy and those that do vary depending on the embeddings used. We urge that unconnected models be used as a baseline for assessing GNN performance.
    \item We provide a new method, named GraNet, that combines GNNs and  models (fine-tuned or trained from scratch) to efficiently exploit the graph structure in raw data.
    \item We empirically show that GraNet outperforms both unconnected models (the strong baseline) and GNNs on a range of datasets and accompanying embeddings.
\end{itemize}

\section{Method}
\label{sec:method}

Our proposed method of converting standard neural network models into graph-connected models blends the two networks. This approach can easily be broken down into individual layers. Taking $f^l_{\bm{\theta}_l}$ as the $l$-th layer in a standard model, $f_{\bm{\theta}}$, where we may be given pre-trained weights $\bm{\theta}$ we can describe this new layer by reformulating \Cref{eq:messagepassing} as such

\begin{equation}
\label{eq:granet}
{\bm{h}}_i^{l} = \gamma_{{\bm{\theta}}_\gamma} \left({\bm{h}}^{l-1}_i,
        \psi_{j \in {\mathbb{N}}(i)} \left( \phi_{{\bm{\theta}}_\phi}
        \left(f^{l-1}_{{\bm{\theta}}_{l-1}} \left( {\bm{h}}^{l-1}_i \right), f^{l-1}_{{\bm{\theta}}_{l-1}} \left( {\bm{h}}^{l-1}_j \right), {\bm{e}}_{j,i}\right) \right) \right)
\end{equation}

where $\mathbf{h}^0_i = \mathbf{x}_i$ rather than applying an embedding function.

\begin{figure}[ht]

  \caption{Graphical representation of how our proposed GraNet layer operates for image networks}
  \centering
  \includegraphics[scale=0.35]{figures/ColorBlind.drawio.pdf}
  \label{fig:GraNet}

\end{figure}

\Cref{fig:GraNet} is a representation of \Cref{eq:granet} specifically in the case where the pre-trained model is a CNN and
($f^{l-1}_{\bm{\theta}_{l-1}}$) is a convolutional layer. The light blue regions perform the standard convolution feature extraction, these extracted feature maps are then adjusted, in the light orange region, by a Message Passing layer with graph attentions. These two representations are then combined.

The light blue regions and resulting channel stacks,
$A_i \in {\mathbb{R}}^{1 \times C_l \times H_l \times W_l}$ through $A_n \in {\mathbb{R}}^{1 \times C_l \times H_l \times W_l}$, represent the forward pass through a single CNN layer $f^{l-1}_{{\bm{\theta}}_{l-1}}$. $A_i$ represents the forward pass of the current node ${\bm{h}}_i$ and $A_{\{j, \dots, n\}}$ represent the forward pass of the neighbours of ${\bm{h}}_i$.
The light orange region and resulting channel stacks, $\mathcal{B}$, represent the graph-based Message Passing stage where the new representations are altered ($\phi_{{\bm{\theta}}_\phi}$), aggregated ($\psi$) and finally combined with the current node representation ($\gamma_{{\bm{\theta}}_\gamma}$), following the description in \Cref{eq:granet}.

\subsection{GraNet for Flickr\_v2}
\label{sec:method:flickr}

Interconnecting every layer in a large pre-trained model is computationally intensive. Therefore, rather than interconnect every single layer in a pre-trained network $f_{{\bm{\theta}}}$ we can graph-connect only the final layer. We thus split the model into two portions: the first set of unconnected layers and the final GraNet layer.

We can therefore look at equation \Cref{eq:granet} and see in this case that we partially carry out the forward pass of $f_{{\bm{\theta}}}$ which we will denote as $f_e$ and then carry out a forward pass through a GraNet layer. This allows us to ignore all the steps involved in applying $f_e$ and focus on the final GraNet layer. Denoting this final GraNet as $g$, and output classification $y$, we achieve the following equation

\begin{equation}
    y = g(f_e({\bm{X}}))
\end{equation}

$f_e({\bm{X}})$ is the same as described in \Cref{sec:related} and indeed if we were to freeze $f_{{\bm{\theta}}}$ this would be equivalent to training GraNet with a single layer on the embeddings created by embedding function $f_e$. So instead we also allow $f_e$ to train thus fine-tuning the weights ${\bm{\theta}}$ of $f_{{\bm{\theta}}}$. This indirectly allows $f_{{\bm{\theta}}}$ to learn the graph structure by providing $g$ with better embeddings.

As GraphSAGE performed the best on ResNet embeddings we use this as our GNN, $g$. As $f_{{\bm{\theta}}_f}$ is already pre-trained but $g_{{\bm{\theta}}_g}$ is not fine-tuning the model produces poor results. We therefore initially train $g_{{\bm{\theta}}_g}$ on a frozen ${f_e}_{{\bm{\theta}}_f}$, after this short training period we unfreeze ${f_e}_{{\bm{\theta}}_f}$ allowing both to train fine-tuning the weights ${\bm{\theta}}_g$ and ${\bm{\theta}}_f$.

\subsection{GraNet for Amazon}
\label{sec:method:amazon}

In the case of the Amazon dataset we found that Bag of Words embedding performed the best. As this does not have an associated pre-trained model we design a multi-layer perceptron (MLP) to compare against. We then convert all the layers within the MLP to GraNet layers. \emph{This method is therefore not fine-tuned but trained from initialised weights.}

We also find that GAT \citep{Brody2021GATv2,Velickovic:2018we} models perform the best on this task. We therefore use graph attention message passing as shown in \Cref{eq:GATv2} for our GraNet model. Keeping in line with \Cref{eq:granet} the new graph attention mechanism becomes

\begin{equation}
    \alpha_{ij} = \frac{exp(\bm{a}^T\mbox{LeakyReLU}([f^l_{{\bm{\theta}}_l}(\bm{h}_i)||f^l_{{\bm{\theta}}_l}(\bm{h}_j)]))}{\sum_{k \in {\mathbb{N}}_i} exp(\bm{a}^T\mbox{LeakyReLU}([f^l_{{\bm{\theta}}_l}(\bm{h}_i)||f^l_{{\bm{\theta}}_l}(\bm{h}_j)]))}
\end{equation}

However, a single layer of an MLP is $f^l_{{\bm{\theta}}_l} \left( {\bm{x}} \right) = {\bm{\theta}}_l {\bm{x}}$. This would therefore mean that \Cref{eq:Gatv3} becomes

\begin{equation}
    \label{eq:Gatv3}
    \alpha_{ij} = \frac{exp({\bm{a}}^T\mbox{LeakyReLU}([{\bm{\theta}}_l{\bm{h}}_i||{\bm{\theta}}_l{\bm{h}}_j]))}{\sum_{k \in {\mathbb{N}}_i} exp({\bm{a}}^T\mbox{LeakyReLU}([{\bm{\theta}}_l{\bm{h}}_i||{\bm{\theta}}_l{\bm{h}}_j]))}
\end{equation}

This is very similar to \Cref{eq:GATv2} with the only difference being when we apply the weight matrix, concatenation and attention mechanism. We find this model behaves the same as GAT2 and therefore, given the small size tried a different approach as a comparison.

Rather than use the vector parameter ${\bm{a}}$ we introduce a linear function $a_{{\bm{\theta}}_a}: {\mathbb{R}}^{2F} \rightarrow {\mathbb{R}}^{F^\prime}$ that takes the LeakyReLU of the concatenation of a node, $f_{{\bm{\theta}}_l}({\bm{h}}_i)$, and its neighbour, $f_{{\bm{\theta}}_l}({\bm{h}}_i)$, as input. The result is a feature vector in a new feature space, though in our case we have $F^\prime = F$. This allows more complex interactions between the node representations to be exploited by our attention. This approach is too costly to apply to a pre-trained CNN as the size of $\bm{\theta_a}$ is far too large.
\section{An In-depth Overview of Existing Datasets}
\label{sec:app:dataset-overview}

Due to the discussed importance of $f_e$ on GNN performance there is a lot to discuss about prior datasets that exist within the space of GNNs in regards to these functions.

\subsection{Pytorch Geometric}
\label{sec:app:pyg}

Pytorch Geometric python library provides a standard interface on top of Pytorch to allow for the development of graph based machine learning. The library also provides a sample of datasets from previous papers published in this field. A brief overview of popular graph-connected datasets, their task and embeddings is available in table \ref{tab:dataset-overview}.

As is clear from the table the current standard embedding for datasets is bag of words. In the cases where bag of words approaches are not used the approach is grounded in classical text representations such as n-grams and word vectors.

The tasks in these popular datasets are node classification where the node data is frequently text. We therefore say that on the node level these are text classification tasks. The only instance of a non-text classification task is Flickr \cite{zeng2019graphsaint}, though based on the fact that the underlying data is image descriptions this could also be considered a text classification task.

This demonstrates how limited the reach of GNNs currently stand as they are being trained on datasets that behave very similarly where the only difference is the specifics of the available data. We feel that this does not therefore fully test the capabilities of GNNs and puts too much emphasis on bag of words and text classification.

\subsection{Open Graph Benchmark}

The results in this paper focus on \textit{node property prediction} as the data that unconnected models ordinarily work on is easily transferred to nodes in a graph. So when discussing Open Graph Benchmark  \citep[OGB]{hu2020open} the focus is on the node property prediction subset (OGBN).

The goal of OGB is to create a standard set of datasets that can be used to compare different GNN architectures so a discussion as to way we did not use their datasets is warranted. The available datasets \textit{ogbn-products, ogbn-proteins, ogbn-arxiv, ogbn-papers100M} and \textit{ogbn-mag} all use variations on the same text representations used in \Cref{sec:app:pyg}. These include Bag of Words (BoW), word2vec and skip-gram. This means the same discussions on these classical text holds here.

We see that the majority of the tasks focus on text classification, excluding \textit{ogbn-protein}, this again draws into question how well these datasets are testing the range of classification tasks. Further to this, focusing mainly on BoW style embeddings raises the question of whether we are building good BoW extractors or graph information extractors.

We do not compare against these datasets as information on how they extracted and washed their data is not readily available and therefore finding matching raw data was not possible. Without the matching raw data we cannot compare a raw version of the dataset to the embedding dataset provided by OGBN.

\subsection{Flickr}
The prior Flickr dataset used in \textit{Zeng et al.} \cite{zeng2019graphsaint} originated from \textit{McAuley et al.} \cite{McAuley2012image} which aimed to utilize network connections and image descriptions rather than the images themselves. The specific embedding function that the paper used is Bag of Words.

This embedding function is a valid representation of images but it is not easily applicable to other image datasets. Thus GNNs trained on this dataset are confined to images with descriptions that have been transformed using the same top 500 words. Noting that this list of top 500 words is not readily available.

\subsection{Amazon}
\textit{Zeng et al.} \cite{zeng2019graphsaint} also provide an Amazon dataset (AmazonProducts) covering the entirety of Amazon. Without a known source we instead use available Amazon databases online to download and generate our own dataset. The embedding function  used is to tokenise the reviews by 4-grams and take the single value decomposition. This is, as with Flickr, not easily applicable outside of the original dataset.

An alternative Amazon dataset (Amazon) is also available from \textit{Shchur et al.} \cite{shchur2018pitfalls} created originally in \textit{McAuley et al.} \cite{McAuley2016image}. Though the original source of the dataset used a pre-trained Caffe model to embed the product images this dataset did not use these. Instead they created their own embeddings using the bag of words standard with the product reviews as the raw data.
\section{Related Work}
\label{sec:related}

\textbf{Graph Augmented Networks} \citet{chen2021on} introduce Graph-augmented MultiLayer Perceptrons (GA-MLPs) as a simplified alternative to Graph Neural Netwroks (GNNs). These models involve a two step process - augmenting the node features of the graph based on the topology and then using these node features applying a learnable function at the node level. This allows a fixed graph operator and two sets of MultiLayer Perceptrons (MLPs) be used to extract features from the graph. This approach is related to similar simplified GNN techniques \citep{wu2019simplifying,nt2019revisiting}. The paper proves that this simplified approach is not as expressive as standard GNNs when looking at the Weisfeiler-Lehman test for distinguishing non-isomorphic graphs. This suggests that GNNs are well suited for inferring information based on graph structure but the paper does not comment on which approach is best in practice. We differ in our approach to augmenting networks with graph structure by using existing GNNs and do not attempt to simplify the network. We do provide a graph-connected MLP but this looks at adding message passing to MLPs rather than separte funcitons on the graph data.

\textbf{Effect of training on GNN performance} \citet{shchur2018pitfalls} look at the effect of hyperparameters and training in GNNs to show that these have dramatic effect on model ordering. Simply changing the split on a dataset caused large changes in accuracy and which GNN performed best, even though the hyperparameters of the GNNs remained constant. We show similar large difference when considering different embeddings with the same splits across embeddings.

\textbf{Ablation studies on GNNs} Further to these discoveries \citet{nt2019revisiting} demonstrate that GNNs only utilise the graph structure to de-noise already highly informative features. They go as far as to demonstrate in certain conditions GNNs and MLPs perform the same. \citet{chen2019powerful} demonstrate that linearising the graph filter stage of GNNs does not hinder but actually increases the performance. Similarly \citet{wu2019simplifying} simplify GNNs by removing non-linearity between layers allow for pre-computing the $k$ message passes. This reduces graph representation learning to a simple linear regression. In all of these cases they demonstrate that the major contribution of GNNs is in their graph structure capabilites. We do not analyse these aspects but look at how this capability can be used in existing unconnected networks.
\section{Reproducibility}

We discuss how we setup all of our experiments in \Cref{sec:setup} including how many runs we completed for each data-point. the specific random seeds that we used for the three runs were 42, 9001 and 27032002. The number of runs for each run is also listed here and for the models that have a different number of epochs the specifics are mentioned in \Cref{sec:app:hyperparameters}.

All test runs were carried out on a single Nvidia A100 80GB GPU, we used pytorch 1.12 with cuda 11.3. All other packages where installed based on these requirements.

\Cref{sec:app:hyperparameters} also contains the specific architectures, samplers, learning rates and learning rate schedulers used in each of the model test runs. These are also linked back to \citet{zeng2019graphsaint} to provide comparable results to the ones achieved in that paper for similar datasets Flickr and Amazon.

\Cref{sec:app:datasets} details the specifics of how we formed our datasets including how we downloaded the raw data, steps we took to wash the data, how we formed the graph and how we created our embeddings. The specifics of how we labelled the data and the labels that we chose are also available.

We have anonymised the dataset including the specific code used to create GraNet models, the config files used and the code to download, wash and build the datasets available \href{https://anonymous.4open.science/r/Revisiting-Embeddings-for-Graph-Neural-Networks}{here}.

\section{Detailed Results}
\label{sec:app:results}

\Cref{tab:amazon-std,tab:instruments-std,tab:flickr-std,tab:granet-std} are the same results as those present in \Cref{sec:eval} but with added standard deviation from the mean.

\input{tables/flickr-std}
\input{tables/instruments-std}
\input{tables/electronics-std}
\input{tables/granet-std}
\section{Experimental Setup}
\label{sec:setup}

For all test results we run 3 train-test runs each with a different random seed and take the arithmetic mean and include the standard deviation. The three random seeds are the same for every entry in the table for fair comparison. The architecture used for each benchmark GNN is identical across all datasets and embeddings. Training takes 300 epochs of training unless otherwise stated.

Where a Graph Neural Network (GNN) is used as part of a GraNet model (as described in \Cref{sec:method}) the same sampler is used for consistency. The specific architectures used are designed using the hyperparameters in \citet{zeng2019graphsaint}. In the case of GraphSAGE the learning rate was decreased to improve convergence. For specific details of each architecture and learning rate see \Cref{sec:app:hyperparameters}.

\subsection{Datasets}

An overview of the datasets is provided below with the specific metrics for each dataset shown in \Cref{tab:flickr-dataset}. A more detailed discussion is available in \Cref{sec:app:datasets}. \emph{It is important to note that though these datasets mirror prior datasets due to the need for raw data we diverge from these datasets}. Therefore we do not make any direct comparisons to previous datasets though the results we achieve on our new datasets are on par with results seen in prior papers.

\subsubsection{Flickr\_v2}
\label{sec:setup:flickr}

The underlying data $\bm{X}$ is raw images and so Convolutional Neural Networks (CNNs) are used as embedding functions. As a sample of existing pre-trained CNNs we use ResNet18, ResNet50 \citep{he2016deep} and VGG16 \citep{simonyan2015very}. In all three cases we use the pre-trained models provided by TorchVision, using the feature vectors after the final pooling stage before the classification stage.

It is important to note that \emph{there is no Bag of Words (BoW) embedding for Flickr\_v2 because there is no sensible object that can be considered a ``word" for raw image data.} The available Flickr \citep{zeng2019graphsaint} uses BoW because the underlying data is image text descriptions not raw images. This limits how a GNN, or general neural network, can classify images as the images must first be processed to provide text descriptions.

\subsubsection{AmazonElectronics and AmazonInstruments}

The underlying data $\bm{X}$ is text so text classification transformers such as RoBERTa \citep{liu2019roberta}
are ideal, specifically we use pre-trained RoBERTa.
We extract three different embeddings from RoBERTa using the pre-trained model provided by fairseq toolkit \citep{ott2019fairseq}. The first is the byte pair tokenisation used by RoBERTa, the second is the feature extraction provided by fairseq which occurs after RoBERTa's transformer heads and before classification, and the final is the feature vector present before the last fully connected layer. Due to restrictions in the token size for RoBERTa we remove all nodes that have reviews with greater than 512 tokens.

We also provide the standard Bag of Words embedding as in the case of text classification this is a common embedding practice, keeping in line with prior datasets \citep{Kipf:2017tc,zeng2019graphsaint}
we use the top 500 words to create our Bag of Word embeddings.
\section{Submission Guidelines}

LoG 2022 solicits papers from areas broadly related to \textbf{machine learning on graphs and geometry}.
Detailed information about the reviewing process is available on the conference website at:
\begin{center}
	\url{https://logconference.org/}
\end{center}
Please follow the guidelines to prepare your submissions to LoG 2022.

\subsection{Important Dates}

Please note the following key dates, all in the \href{https://www.timeanddate.com/time/zones/aoe}{Anywhere on Earth (AoE)} timezone:
\begin{itemize}
	\item September 9, 2022: Abstract submission deadline
	\item September 16, 2022: Final submission deadline
	\item October 20, 2022: Two-week paper revision period starts
	\item November 3, 2022: Paper revision period ends
	\item November 24, 2022: Final decisions released
	\item November 30, 2022: Camera ready submission deadline
	\item December 9, 2022: Conference starts
	\item December 12, 2022: Conference concludes
\end{itemize}

\subsection{Dual-Track Submission}

The LoG conference has a proceedings track with papers published in \href{https://proceedings.mlr.press/}{Proceedings for Machine Learning Research (PMLR)} series and a non-archival extended abstract track.

For the proceedings track, papers may only be up to \textbf{nine} pages long.
Extended abstracts can only be up to \textbf{four} pages long.
Additionally, \textbf{unlimited} pages \emph{containing only acknowledgments, references, and appendices} are allowed.
Submissions that exceed the above page limits will be desk rejected.

\subsubsection{The Proceedings Track}
Accepted proceedings papers will be published in the \href{https://proceedings.mlr.press/}{Proceedings for Machine Learning Research (PMLR)} series and are eligible for our proceedings spotlights.
Submitted papers cannot be already published or under review in any other archival venue.
Upon acceptance of a paper, at least one of the authors must join the conference, or their paper will not be included in the proceedings.

\subsubsection{The Extended Abstract Track}
The top accepted extended abstracts are chosen for our abstract spotlights.
Authors of accepted extended abstracts (non-archival submissions) retain full copyright of their work.
Acceptance to LoG does not preclude publication of the same material at another venue.
Also, submissions that are under review or have been recently published are allowed for submission to this track.
However, authors must ensure that they are not violating dual submission policies of the other venue.

\subsection{Double-Blind Reviewing}
Submissions will be double-blind: reviewers cannot see author names when conducting reviews and authors cannot see reviewer names.
We use \href{https://openreview.net/group?id=logconference.io/LOG/2022/Conference}{OpenReview} to host papers and allow for public discussions that can be seen by all; comments that are posted by reviewers will remain anonymous.

\section{Formatting Instructions}

All submissions must adhere to the following formatting specifications.

\subsection{Style Files}

The only supported style file for LoG 2022 is \verb+log_2022.sty+ with \LaTeX.
The initial submission and the camera ready versions have to be submitted as \texttt{.pdf} files.

By default, the style file creates camera ready versions for accepted proceedings track papers, where the footnote in the first page gives the full reference of the paper.
The style file contains three additional options:
\begin{itemize}
	\item \verb+[review]+ anonymizes the manuscript for double-blind review. It also adds line numbers to help reviewers navigate to particular lines.
	\item \verb+[preprint]+ creates non-anonymous versions for preprint platforms (e.g., arXiv), with the text ``Preprint. Preliminary work.'' in the footnote.
	\item \verb+[eatrack]+ creates camera ready versions for accepted extended abstracts with the footnote containing pertinent paper information.
\end{itemize}

\subsection{Layout}
Manuscripts must be set in one-column US letter papers.
All text should be confined within a rectangle 5.5-inch wide and 9-inch long.
The left and right margins are both 1.5 inches.
All pages should start at 1 inch from the top of the page.

\subsection{Font}
All text (except listings, non-Latin scripts, and mathematical formulas) should use the \textbf{Times New Roman} typeface.
\textbf{Computer Modern} fonts are preferred for mathematical symbols and formulas.
Furthermore, please make sure your PDF file only contains Type-1 fonts.

\subsection{Title}

The paper title should be 14-point bold type and centered between two horizontal rules that are both 1-point thick.
There is a 1/4-inch space above and below the title to rules.
Make sure that the title is in Initial Caps --- The First Letter of Content Words Should be Made Capital Letters.

Besides, the title will display as the running head on each page except the first one.
The running title consists of a single line in 9-point font, centered above a horizontal rule, that is 1/2-point thick and 20-point above the main text.
The original title is automatically set as the running head.
In case that the original title exceeds one line, a shorter form can be specified by using \verb+\title[A Shorter Title]{A Long Title}+.

\subsection{Author Information}
The author information appears immediately after the title, with a 24-point space in between.
Submissions to LoG 2022 should be properly anonymized. Once the \verb+[review]+ option is passed, the author information will not be printed.

For the final version, author names are set in 10-point bold type, institution names and addresses are in 9-font, and email addresses are in 9-point typewriter font.
Each name should be centered above the corresponding institution and email address.
Authors can use \verb+\institute+ and \verb+\email+ to specify affiliations and email addresses respectively.

To ensure that the reference information in the bottom of the first page is rendered appropriately and concisely, please specify an abbreviated author list by shortening the first name of the first author in initials followed by ``et al.'': \verb+\author[Y. Zhu et al.]{Yanqiao Zhu \and Yuanqi Du}+.

With the provided style file, the author information can be set in various styles.
If all authors are from the same institution, authors can use:
\begin{quote}
	\begin{verbatim}
		\author[F. Last et al.]
		 {First Last 1, First Last 2, ... \and First Last n \\
		  \institute{Institute \\ Address line}\\
		  \email{first.last@example.com}}
	\end{verbatim}
\end{quote}
For authors from different institutions, please use the \verb+\And+ command:
\begin{quote}
	\begin{verbatim}
		\author[F. Last et al.]
		 {First Last 1 \\
		  \institute{Institute 1\\ Address line} \\
		  \email{first.last.1@example.com}\And
		  First Last 2 \\
		  \institute{Institute 2\\ Address line} \\
		  \email{first.last.2@example.com}}
	\end{verbatim}
\end{quote}
If author names do not fit in one line, use the \verb+\AND+ command to start a separate row of authors:
\begin{quote}
	\begin{verbatim}
		\author[F. Last et al.]
		 {First Last 1, ..., \and First Last n \\
		  \institute{Institute 1\\ Address line} \\
		  \email{first.last.1@example.com}\AND
		  First Last n+1 \\
		  \institute{Institute 2\\ Address line} \\
		  \email{first.last.n.1@example.com}}
	\end{verbatim}
\end{quote}

\subsection{Abstract}

The paper abstract should be placed 0.2 inches below the final address.
The heading word ``Abstract'' should be centered, bold, and in 10-point type.
The abstract body should use 10-point type with a vertical spacing of 11 points and be indented 0.5 inches on both the left- and right-hand margins.

\subsection{Headings and Sections}

All section headings should be numbered, flush left, and bold with content words capitalized.
\begin{itemize}
	\item First-level headings should be in 12-point type. Leave a 12-point space before and a 2-point space after the heading.
	\item Second-level headings should be in 10-point type. Leave a 8-point space before the heading and a 2-point space afterward.
	\item Third-level headings should be in 10-point type. Leave a 6-point space before and a 2-point space after the heading.
\end{itemize}
Please use no more than three levels of headings.

\paragraph{Paragraphs}
There is also a \verb+\paragraph+ command available, which sets the heading in bold, flush left, and inline with the text.
Leave a 6-point vertical space before the heading and 1em of horizontal space following the heading.

\paragraph{Footnotes}
Use footnotes to provide readers with additional information about a topic without interrupting the flow of the paper.
Footnotes should be numbered sequentially and placed in 9-point type at the bottom of the page on which they appear.
Precede the footnotes with a horizontal rule of 2 inches.
Note that footnotes should be properly typeset \emph{after} punctuation marks.\footnote{This is an example footnote.}

\subsection{Figures}

\begin{figure}
	\centering
	\includegraphics[width=0.5\linewidth]{example-image-duck}
	\caption{A sample figure.}
	\label{fig:sample}
\end{figure}

All included artwork must be neat, legible, and separated from the text.
The figure number and caption always appear below the figure.
The figure label should be in boldface and numbered consecutively.
The caption should be set in 9-point type, in sentence case, and centered unless it runs more than one lines, in which case it should be flush left.
See Figure~\ref{fig:sample} for an example.

\subsection{Tables}

\begin{table}
	\centering
	\caption{A sample table.}
	\label{tab:sample}
	\begin{tabular}{ccccc}
		\toprule
		\multirow{2.5}{*}{Method} & \multicolumn{2}{c}{Data 1} & \multicolumn{2}{c}{Data 2}  \\
		\cmidrule(lr){2-3} \cmidrule(lr){4-5}
		& \(\mathbf{X}\) & \(\mathbf{Y}\) & \(\mathbf{X}\) & \(\mathbf{Y}\) \\
		\midrule
		A & 0.8817  & 0.9572 & 0.1893 & 0.1725 \\
		B & 0.7126  & 0.2615 & 0.9173 & 0.1286 \\
		C & 0.2716  & 0.1826 & 0.2836 & 0.1836 \\
		\bottomrule
	\end{tabular}
\end{table}

Like figures, tables should be legible and numbered consecutively.
However, the table number and caption should always appear above the table.
See Table~\ref{tab:sample} for an example.

Note that publication-quality tables \emph{do not contain vertical rules and double rules}.
We strongly suggest the use of the \texttt{booktabs} package which provides the commands \verb+\toprule+, \verb+\midrule+, and \verb+\bottomrule+ to enhance the quality of tables.

\subsection{Paragraphs}

Paragraphs are separated by 1/2 line space (5.5 points).
Do not indent the first line of a given paragraph.

\subsection{Equations}

The provided style file loads the \verb+amsmath+ package automatically.
Unnumbered single-lined equations should be displayed using \verb+\[+ and \verb+\]+. For example:
\[
	\mathbf{X}' = \sigma(\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}} \mathbf{XW}).
\]
Numbered single-line equations should be displayed using the \verb+equation+ environment. For example:
\begin{equation}
	\mathbf{X}' = \sigma(\widetilde{\mathbf{D}}^{-1}\widetilde{\mathbf{A}}\mathbf{XW}).
\end{equation}

\subsection{Bibliographies}

Use an unnumbered first-level heading for the references.
For a citation, use \verb+\cite+, e.g.,~\cite{Kipf:2017tc}.
For a textual citation, use \verb+\citet+, e.g.,~\citet{Velickovic:2018we}.
\emph{Any choice} of citation style is allowed as long as it is used consistently throughout the whole paper.
Additionally, both \verb+natbib+ and \verb+bibLaTeX+ packages are supported.
It is also possible to reduce the font size to \verb+\small+ (9-point font) when listing the references.

In the submission version, authors should refer to their own work in the third person for blind review.
In particular, avoid phrases that may reveal personal identities (e.g., ``in our earlier work~\cite{Hamilton:2017tp}, we have shown ...'').

\section*{Author Contributions}
Authors of accepted papers are \emph{encouraged} to include a statement that declares the individual contribution of every author, especially when there are co-authors that made equal contributions to the research.
You may adopt the \href{https://credit.niso.org/}{Contributor Roles Taxonomy (CRediT)} methodology for attributing contributions.
Do not include this section in the version for blind review.
This section does not count towards the page limit.

\section*{Acknowledgements}

The \LaTeX{} template of LoG 2022 is heavily borrowed from NeurIPS 2022.

Do not include acknowledgements in the version for blind review.
If a paper is accepted, please place such acknowledgements in an unnumbered section at the end of the paper, immediately before the references.
The acknowledgements do not count towards the page limit.

redpajama_set_name: RedPajamaArXiv


Reetdachhof-Haus-Lotte-304821 is a property located in Utersum near Amrumer Odde. Guests can enjoy the free WiFi. The holiday home comes with a TV. The kitchenette has a dishwasher and an oven, as well as coffee machine.


ISO/IEC 20000 is the first international standard for IT Service Management (ITSM), and is the only audit-able International Standard which defines the requirements for an IT Service Management system. The standard promotes the adoption of an integrated process approach to effectively deliver managed services to meet the business and customer requirements. In a global marketplace, formal standards provide a common and consistent platform for organisations to work towards.
Any organisation that wants to be formally certified against the scheme will need to be assessed by an itSMF Registered Certification Body (RCB). Once the requirements of ISO/IEC 20000 have been satisfied, the RCB will issue a certificate of conformance and the organisation will be eligible to use the itSMF ISO/IEC 20000 logo as a sign of their achievement.
Pink Elephant's ISO20000 Readiness Assessment is designed to be an objective survey of the level of readiness of an organisation for the formal certification process, and to measure the Quality of your Service Management Processes against the requirements defined by the ISO\IEC20000 international standard in Service Management. This is a useful pre-audit exercise which highlights areas of non-conformance which need to be addressed to achieve formal accreditation.
Through structured interviews with operational and strategic personnel, we build a view of your IT service management group and how the activities they undertake align with those defined as required by ISO/IEC20000, after which we perform a Gap Analysis to highlight priority areas as mandated by the standard. This gives you the opportunity to direct the focus on IT to the areas of the standard that require redressing before formal assessment and certification are undertaken.
Most organisations already have a number of information security policies and controls in place, yet, these tend to be fragmented and are often based on generic threats or past security incidents.
A formal Readiness Assessment is not a requirement of certification to the ISO/IEC 2700 Standard but it can be helpful in assisting your organisation to prepare for initial certification.
The intention of the assessment is to save the organisation time and money by identifying deficiencies in its Information Security Management System (ISMS) before seeking Certification to the ISO/IEC 27001 Standard.
Pink Elephant consultants have been helping customers comply with State and Federal business and privacy regulations for more than a decade. Working as either a full-service consultant, or as an adjunct to your in-house teams, Pink Elephant will execute our phased compliance readiness process to ensure that the business meets or exceeds your compliance requirements.
Creating a comprehensive information security policy.
Performing an audit to determine current level of regulatory compliance.
Providing remediation for vulnerabilities detected on your systems.
Advising your company on specific steps needed to achieve compliance.
Deploying security infrastructure to encrypt email messages automatically.
Encrypting your company's laptops and other mobile devices.
Securing your primary security infrastructure, including firewalls, VPN access, anti-phishing, and tools to protect against malicious code.
VISION: WHERE DO WE WANT TO BE?
ISO 27001 ASSESSMENT: WHERE ARE WE NOW?
PLAN: HOW WILL WE GET THERE?
BUILD & TRANSITION: HOW WILL WE REALISE OUR VISION?
CONTROL: HOW CAN WE KEEP YOU ON TRACK?
Certified compliance with ISO/IEC 27001 by an accredited and respected certification body is entirely optional but is increasingly being demanded from suppliers and business partners by organisations that are (quite rightly!) concerned about the security of their information, and about information security throughout the supply chain or network.
Request a quotation or more information.


text:
The Gary Stewart Show
Buffalo Wild Wings Classic
Small College Basketball
DIII News
Hero Sports
Mustangs Close Out Season at Alvernia Tuesday
Stevenson (11-13, 3-12 MAC Commonwealth) vs.
Alvernia (11-12, 6-8 MAC Commonwealth)
Tuesday, February 13, 2018 - 8 pm
Reading, Pa. - McCloskey Court (1,500)
COMPLETE GAME NOTES (PDF)
STORY LINES
Senior Danzel Wright is one of 100 players nationally named to the Bevo Francis Award watch list, given to the top player in small college basketball.
Wright ranks second in the MAC Commonwealth in scoring (20.4 ppg), third in three-point percentage (.408) and second in free throw percentage (.898) while ranking ninth in field goal percentage (.453)
Stevenson scored 113 points in a November 25 win at Penn State Berks, matching a program record for points in a game.
After winning just four games a season ago, the Mustangs have 11 victories in 2017-18.
Stevenson's five-game win streak earlier this season was its longest since winning seven straight from January 14-February 7, 2015.
Saturday's win over No. 16 Lycoming was the Mustangs' first over a ranked opponent since downing No. 25 Messiah on February 12, 2014.
• Alvernia holds an 9-4 lead in the series since Stevenson joined the MAC Commonwealth in 2012-13. Included in that total are two Alvernia wins in the conference title game in 2014 and 2015. In the first meeting this season, Stevenson trailed by as many as 27 in the second half before coming back to fall by two, 85-83
• Senior Danzel Wright has made an immediate impact in his first season at Stevenson. Reaching double figures in scoring in every game but one, Wright ranks among the national leaders in scoring (20.4 ppg), total three-pointers made (78) and free throw percentage (.898). Wright was named to the Bevo Francis Award watch list on January 15, one of only 100, for the award that honors the top player in all of small college basketball.
WORKING OVERTIME
• Entering the season, the Mustangs had not been to overtime since February 7, 2015 against Messiah, a span of 60 games, but Stevenson has been to an extra session twice already in 2017-18. In the season opener, it outlasted Elizabethtown 90-87, while on December 2, the Mustangs fell 78-75 at home in OT to Widener.
PILING UP THE POINTS
• After scoring at least 80 points just four times last season, Stevenson has cracked the 80-point mark 10 times, going 7-3 in those games. The Mustangs have gone over the 90-point mark three times, including a 113-point outburst at Penn State Berks on November 25 that tied the program record for points in a game.
IN A GOOD MOODY
• Junior Keith Moody has reached double figures in five of the past seven games after doing so just twice in the first 17 games of the season. He is averaging 9.7 ppg over the last seven games, including a career-best 17 against Widener last Saturday. Moody also posted his first double-double of the season against Lycoming with 10 points and 13 rebounds.
PULLING RANK
• By defeating No. 16 Lycoming on Saturday, it was the first time the Mustangs topped a ranked opponent since downing No. 25 Messiah on February 12, 2014. Stevenson was 3-0 against ranked foes in that 2013-14 season.
ALMOST ALL THE WAY BACK
• Stevenson trailed Alvernia 56-29 with 16:57 remaining on January 13 before storming nearly all the way back in an eventual 85-83 defeat, outscoring the Golden Wolves 54-29 the rest of the way. The following game, the Mustangs trailed 42-22 at the break at Hood and got the Blazer lead down to five, 77-72, with 2:41 left.
• Stevenson athletics can be found on Facebook by searching for "Stevenson University Mustang Athletics" or on Twitter, Instagram and Snapchat at @GoMustangSports. Stevenson men's basketball can be found on Twitter at @GoMustangHoops.
• All Stevenson men's basketball home games can be seen and heard on the Mustang Sports Network. To watch, simply visit www.gomustangsports.tv, or visit the Stevenson University Athletics Roku channel. To listen to the broadcast of any home game, simply visit sfmsports.net.
The Gary Stewart Show presented by Buffalo Wild Wings, in which Stevenson head coach Gary Stewart discusses the past week's action and looks forward to the week ahead, can be seen each Tuesday at GoMustangSports.com as well as on Roku.
Follow Stevenson University athletics on Twitter, Facebook, Instagram, Youtube and Snapchat
Follow Stevenson men's basketball on Twitter
Sign up to receive email news on your favorite Stevenson teams here
Wed, 02/27 | Men's Basketball at Rutgers-Newark L, 77-74 (Final) RC | BX
Mon, 02/18 | Men's Basketball vs. Albright L, 78-77 (Final) RC | BX | PH | V
Sat, 02/16 | Men's Basketball at Albright L, 75-69 (Final) RC | BX
Tue, 02/12 | Men's Basketball vs. Hood L, 90-74 (Final) RC | BX | PH | V
Sat, 02/09 | Men's Basketball at Widener L, 75-55 (Final) RC | BX
Wed, 02/06 | Men's Basketball vs. Lebanon Valley L, 85-75 (Final) RC | BX | PH | V
Sat, 02/02 | Men's Basketball at Lycoming W, 81-57 (Final) RC | BX | V
Wed, 01/30 | Men's Basketball at Messiah W, 86-84 (Final - OT) RC | BX
Sat, 01/26 | Men's Basketball vs. Arcadia L, 79-78 (Final) RC | BX
Wed, 01/23 | Men's Basketball vs. Alvernia L, 87-73 (Final) RC | BX | PH | V

redpajama_set_name: RedPajamaCommonCrawl


Oster Tanz Workshops is a diverse program accessible to different levels and age groups and easy to combine with each other.
In April from 23.-27.04, Oster Tanz will continue to thrill you with unusual metaphors on geometric forms of a butterfly, awakening bodies in dynamic states, connections between emotion and movement at the limits of transforming potentials or even lyrical exploration of movement. Have fun exploring!
Next, to the workshops, we will also have the format Learning. Teaching. Performing. again, but this year it will be particularly attuned to the participants' perspective of learning and experiencing and we want to cordially invite you all to come and share your perspective.


Provides funding to school boards to offset the costs of equipment essential to support individual students.
The equipment is to provide students with accommodations essential to access the Ontario curriculum and/or Board-determined alternative program and/or to attend school.
ensure that equipment is transferred to another student when it is not required by the student for whom it was purchased.


text:
News: 18 December 2018Planning permission secured for a new community in Melksham
Our Architecture and Bristol Planning team have secured detailed planning permission for 447 homes in Melksham, Wiltshire on behalf of Bloor Homes and Hallam Land. The scheme includes a new link road that runs along the southern edge of Melksham which will connect the new development to the Clackers Brook development to the north and Spa Road to the east.
The development layout has retained almost all of the landscape hedgerows on-site and provides a generous area of open space in the centre of the scheme, creating a walkable green network of pedestrian and cycle links throughout the site. The design has also responded sensitively to Bower Hill Lodge Farm, a listed building located adjacent to the site, by providing a wide landscape margin against the western boundaries to facilitate new tree and hedgerow planting.
Design Director Jacques Toerien said: "By working alongside the Urban Design Officer, we were able to revise the designs, focussing on positive place making by putting people and landscape first. This led to a development that is well-connected with the surrounding area and provides positive frontage and outlook to the north, west and south of the site while respecting the setting of Bower Hill Lodge Farm to the east"
Planning Director Peter Roberts said: "We are delighted that Planning Permission has been granted. Working in partnership with the Local Authority and Urban Design Officer has been key to ensuring this is the correct scheme for the area."
Construction will begin on-site in the new year and 447 homes in total will be built.
UK Housing, New Communities, Design, Planning
Jacques Toerien
Josh Kittlety
Associate Architectural Technician
Sector » New Communities
News » Planning submitted for a sustainable new community in Bishop's Stortford
Our team have submitted a Hybrid Planning Application for a...
News » Green light given for Harworth South Business Park
Thousands of new jobs are expected to be created as councillors voted...
Blog » From retail to resi – putting the life back into city centres
Dan Mitchell blogs...
Blog » Empowering the masses - not the few - in Scotland
Colin Lavety blogs...
News » Major new employment area proposed for Carnforth
An additional 90,000 sqft of employment space could be coming to...
News » We've been shortlisted...
We've been shortlisted, with KC+A, for the London Borough of Hounslow...
Blog » Wales Planning Bill: latest stage brings positive news
Pete Sulley blogs...


'Chains' is a short film about growing up and growing apart. In the early hours of the morning, after an insane house party, two old friends bump into each other. As they wait for the first bus they reminisce over the good times they shared and eventually talk about why their friendship ended.
Frankie is a young woman with shoulder length/long brown hair. She dresses in mostly dark clothing. Although comfortable with herself and confident she is quiet and doesn't enjoy being the center of attention or engaging in conflict. A wallflower.
Rachel is a young outgoing woman with blonde hair. She loves to be the center of attention and is confident in herself. She enjoys partying and says what she thinks, she has no filter. Rachel is a loyal friend.
Looking for fifteen to twenty people for a house party scene.
Must be based in Dublin.
Please email orlaghjdoherty@gmail.com if interested.
If applying for the roles of Frankie or Rachel please provide headshots, showreel (if possible) and a brief description of past work.
If applying as an extra please send headshots and pictures of your outfit of choice if you were to attend a house party.
Unfortunately this is a student production therefore there will be no payment available however all cast will have access to clips for their showreel.


BUZZARDS BAY, Mass. - Sophomore striker Ben Cook (Ft. Myers, Fla./Ft. Myers) tallied twice in the opening half of Saturday's Men's Soccer season opener as Massachusetts Maritime (1-0-0) posted a 2-1 non-conference victory over the visiting Rams of Suffolk University (0-1-0) at Clean Harbors Stadium.
Cook delivered the opening blow in the 12th minute on a breakaway, slipping around the Suffolk keeper to the left side. Eight minutes later Cook put the Buccaneers on top with his second goal of the afternoon with a hard shot from 20-yards out that drilled the back of the net.
The Buc's continued to pressure Suffolk keeper David Haines (New York, NY/Dwight School) as Andrew Wayrynen (Rockport, Mass. /Rockport) launched a header off a cross from Cook that forced Haines to make a leaping save. Brady Lemmon (California, MD/St. Mary's Ryken) slammed a ball off the post and Jacob Crawford (Rockland, Mass./Rockland) launched a header off a corner shot by Zach O'Neil (Scituate, Mass./Scituate) as the Buc's kept the heat on Haines in the Suffolk net.
Andrew Young (Glasgow Scotland/Gems World Academy) broke up a Massachusetts run in the 41st minute and Declan Mee (Westwood, Mass./Westwood) got off a shot at the Buccaneers net two minutes later that sailed high.
The Buccaneers took their two-goal lead into the second half facing increased pressure up top from the Rams. Jordan Casey (East Hampton, Conn./East Hampton) and Andrew Collari (Leicester, Mass./Wethersfield) led a vigorous Suffolk attack in the second testing the Buccaneer back line of Ben Paine (Dartmouth, Mass./Dartmouth), Nick Van Ness (Waldwick, NJ/Waldwick), Kyle Daly (Plymouth, Mass./Plymouth South) and Jake Pforr (Weymouth, Mass./Weymouth), who fought off the Ram attack until with 20 minutes to play Tucker Mee ( Westwood, Mass./Westwood) took a ball from Leonardo Acosta (Miami, Fla./Miami Beach Senior) and fired a rocket past the outstretched reach of MMA keeper Mitch Hardy (Pepperell, Mass./Groton-Dunstable).
The Buccaneers outshot the Rams 14-4 in the opening half, while the Rams came back to take a 10-4 advantage in shots in the second.
Massachusetts Maritime will host Newbury College on Monday at Clean Harbors Stadium at 6 PM, while the Rams will host Framingham State Monday at noon.


Ee Ma Yau. This is the title of Lijo Jose Pellissery's new film. Very famous for experimental subjects and different style of treatment, Lijo is back after his most successful Angamaly Diaries. Ee Ma Yau is a satire based on death and related incidents.
has Vinayakan, Chemban Vinod and Dileesh Pothen in the lead characters. The casting itself created enough discussions.
Script is penned by PF Mathews, the national award winner and the writer of Iruttil Oru Punyalan. The shoot is over and the film is planned for December release.
Jeethu Joseph Next With Prithviraj.


The fact that cancer and its treatment negatively affect patient sleep is well known. However, whether sleep disturbances could enhance cancer development and progression is a question that has been raised more recently. There is, however, considerable evidence from epidemiological and clinical studies and from of experimental research in cell and animal models strongly suggesting that sleep alterations may increase the growth and metastasis of several types of malignancies. Different sleep disturbances can boost cancer growth. On the one hand, changes in sleep architecture, for instance modification of the circadian cycle by shift work or sleep fragmentation caused by sleep apnea). On the other hand, the events of chronic intermittent hypoxia experienced by patients with sleep apnea have also been shown to increase cancer progression. Current basic and clinical research is focused on identifying the pathophysiological mechanisms determining the interaction between sleep and cancer.
Ramón Farré (RF) is Professor of Physiology at the Biophysics and Bioengineering Unit of the Faculty of Medicine and Health Sciences of the University of Barcelona (UB) since 2003. In the teaching field, he has been taking responsibilities of coordination in the Degrees of Medicine and Biomedical Engineering and in the Master of Bioengineering. In the field of university management, RF has been Secretary of the Department of Physiological Sciences I, Secretary and Vice-Dean of the Faculty of Medicine and member of the Delegate Economic Commission of the Governing Council of the UB. In the field of national scientific management, RF is Coordinator of the Area "Biopathology and respiratory, cardiovascular and renal bioengineering" of the Biomedical Research Institute of August Pi i Sunyer (IDIBAPS), is head of the research group "Respiratory biophysics and bioengineering" of the IDIBAPS, and head of Group 12 of the CIBER of Respiratory Diseases. At the international level, in the last 10 years RF has been "officer" of the "Clinical Respiratory Physiology, Exercise and Functional Imaging" Assembly of the European Respiratory Society, and has been a member of the strategic planning of the "Sleep and Respiratory Neurobiology" Assembly of the American Thoracic Society. Since 2008 he is Associate Editor of the European Respiratory Journal (1st decile ISI). Since the creation of EIT-Health (2015), RF is Director of Education of its Spanish node and is a member of the "Strategic Committee of Education" of EIT-Health (Munich, Germany). RF research is located in the field of bioengineering. Specifically, it is aimed at deepening our understanding of the mechanical behavior of the respiratory system to improve the diagnosis and treatment of respiratory diseases. He has published 230 international peer-reviewed papers (h-index=40).


text:
For Buckeye Nation, the current playoff system giveth and taketh away.
By Mac Aljancic Times-Reporter sports correspondent
In high school, college, and professional athletics, the outcome of all championships are determined entirely on the field of play.
The regular season weeds out the pretenders. The post-season then pits worthy qualifiers face-to-face in elimination clashes that end when only the one true champion is left standing.
The glaring exception to this format, however, is the beauty pageant of college football. For well over a century, national championships have been awarded through an inexact recipe of performance, perception, and opinions. Teams play completely different schedules and different styles of football throughout the season in their own conference territories. From the results of those thousands of games, a group of humans then make a final decision as to "who's the fairest of them all."
For most of the 20th century, human opinion was the sole judge and jury in awarding the crown. In 1968, Associated Press voters declared that 10-0 Ohio State was better than the 11-0 Penn State. In 1978, those voters declared that 11-1 Alabama was better than 11-1 Oklahoma and 12-1 USC.
The BCS system arrived in 1998, allowing humans (sometimes with computer help) to anoint two beauty queens to fight for the crown. Often times, multiple teams were deserving of those precious two spots. The 2003 BCS system chose 12-1 Oklahoma (which lost their conference championship game 35-7) for the championship dance spot over 12-1 USC. In 2007, 12-1 Florida crushed undefeated Ohio State for the crown. But prior to that game, many felt that 12-1 Michigan was more worthy of a championship bid than the Gators.
In 2014, that legendary Dr. Pepper vendor, Larry Culpepper, invented the current four-team playoff, which has been an enormous hit. Champions have been decided more on the ball field and less in the ballot box. But unfortunately, in each year has had a fifth or sixth team feeling it was snubbed.'
The Buckeyes have been on both sides of the coin for all four years of the current format. The 2014 squad suffered an ugly opening-game home loss to Virginia Tech, and looked to be on the outs. But a 59-0 romp over Wisconsin in the Big Ten championship game jumped 12-1 OSU to the No. 4 spot, just ahead of the Big 12 co-champions, 11-1 Baylor and 11-1 TCU.
The powerful 2015 Buckeyes lost their only game on the final play in Week 11 to Michigan State. The Spartans, along with Alabama and Oklahoma, all made the playoffs with both one loss and a conference championship trophy. Ohio State actually finished No. 7 on Selection Sunday behind 12-1 Iowa and 11-2 PAC-12 champ Stanford. But in an eye test, OSU was surely a top-four team.
Last year, the Buckeyes sole loss at Penn State cost them a chance at a Big Ten Championship. But the playoff committee set a precedent by making OSU the first non-conference champion to earn a playoff berth, due to their strong resume of wins. The two-loss Nittany Lions felt that their Big Ten title and head-to-head win should have trumped the Buckeyes.
Last year's precedent came back to bite this year's Ohio State team in the butt. Did 11-1 Alabama, which did not make the SEC championship game, deserve to be chosen for the No. 4 spot over the 11-2 Big Ten champion Buckeyes? I believe part of that decision was due to the bias towards the "mystique" of Nick Saban's dynasty that has won four national titles in the last eight years. However, I regretfully have to agree with the committee's choice.
Ohio State's 31-point loss in November to a 7-5 Iowa squad, on top of an ugly 31-16 home loss to Oklahoma, made for an ugly scar that was too hard to ignore. Alabama's schedule lacked the quality wins of OSU. But at the beginning of the season, nobody saw the Crimson Tide's schedule, which opened against then No. 3 Florida State an included the powers of the SEC West, as a detriment.
For Buckeye Nation, the current playoff system giveth and taketh away. But overall, this playoff system has been a blast in comparison to its predecessors. The question is, if four teams is good, would eight teams be great? I'll explore that topic in an upcoming article.


We are happy to announce the official release of Open GEE 5.2.4! This release updates Open GEE with minor enhancements and bug fixes for GEE Fusion and Server.
Progress indicators during gesystemmanager processing. Added progress log messages to gesystemmanager while assets are being built.
New "Save and Build" file menu option. Fusion UI now includes a new file menu option that allows users to save and build assets with one click, instead of clicking the save option and then the build option immediately after.
C++98 no longer supported. Building open GEE using C++98 standard is deprecated and is no longer supported. C++11 syntax can be used.
New build parameter to redirect the build output. Build output can now be optionally directed to a folder of your choosing instead of the current hard-coded defaults. See earth_enterprise/BUILD.md in source code for more details.
New experimental build cache folder parameter to the build. This is an experimental option to potentially speed up builds. More work remains to be done before using this option in production builds. Use of this option is not recommended except for giving feedback and/or helping to address issues with Open GEE's build process using scons cache. See earth_enterprise/BUILD.md in source code for more details.
Our thanks go out to all of the contributors who helped make this release possible! The next release, Open GEE 5.2.5, is already underway!