You will be shown sections from a scientific research paper, together with a question about the paper. Your task is to answer the question based on the contents of the paper. Your response should be a `json` object with two fields:

"answer": An array of strings extracted directly from the paper which, collectively, answer the question.
"evidence": An array of strings. Each should be an excerpt from the paper, in which one or more of the extracted answers can be found.

For instance, for the question "What baselines did the authors compare against?", a sample response might be:

{
  "answer": ["BERT", "RoBERTa"],
  "evidence": ["In our experiments, we compare the performance of our model against BERT and RoBERTa."]
}

Do not include any text in your response other than the json.

If the question is unanswerable given the provided excerpts, respond with the single word "null".

Paper: Knowledge Graph Representation with Jointly Structural and Textual Encoding

The objective of knowledge graph embedding is to encode both entities and relations of knowledge graphs into continuous low-dimensional vector spaces. Previously, most works focused on symbolic representation of knowledge graph with structure information, which can not handle new entities or entities with few facts well. In this paper, we propose a novel deep architecture to utilize both structural and textual information of entities. Specifically, we introduce three neural models to encode the valuable information from text description of entity, among which an attentive model can select related information as needed. Then, a gating mechanism is applied to integrate representations of structure and text into a unified architecture. Experiments show that our models outperform baseline by margin on link prediction and triplet classification tasks. Source codes of this paper will be available on Github.

Knowledge Graph Embedding
In this section, we briefly introduce the background knowledge about the knowledge graph embedding.
Knowledge graph embedding aims to model multi-relational data (entities and relations) into a continuous low-dimensional vector space. Given a pair of entities $(h,t)$ and their relation $r$ , we can represent them with a triple $(h,r,t)$ . A score function $f(h,r, t)$ is defined to model the correctness of the triple $(h,r,t)$ , thus to distinguish whether two entities $h$ and $t$ are in a certain relationship $r$ . $f(h,r, t)$ should be larger for a golden triplet $(h, r, t)$ that corresponds to a true fact in real world, otherwise $r$0 should be lower for an negative triplet.
The difference among the existing methods varies between linear BIBREF2 , BIBREF8 and nonlinear BIBREF3 score functions in the low-dimensional vector space.
Among these methods, TransE BIBREF2 is a simple and effective approach, which learns the vector embeddings for both entities and relationships. Its basic idea is that the relationship between two entities is supposed to correspond to a translation between the embeddings of entities, that is, $\textbf {h}+ \mathbf {r}\approx \mathbf {t}$ when $(h,r,t)$ holds.
TransE's score function is defined as: 
$$f(h,r,t)) &= -\Vert \textbf {h}+\mathbf {r}-\mathbf {t}\Vert _{2}^2$$   (Eq. 5) 
 where $\textbf {h},\mathbf {t},\mathbf {r}\in \mathbb {R}^d$ are embeddings of $h,t,r$ respectively, and satisfy $\Vert \textbf {h}\Vert ^2_2=\Vert \mathbf {t}\Vert ^2_2=1$ . The $\textbf {h}, \mathbf {r}, \mathbf {t}$ are indexed by a lookup table respectively.

Neural Text Encoding
Given an entity in most of the existing knowledge bases, there is always an available corresponding text description with valuable semantic information for this entity, which can provide beneficial supplement for entity representation.
To encode the representation of a entity from its text description, we need to encode the variable-length sentence to a fixed-length vector. There are several kinds of neural models used in sentence modeling. These models generally consist of a projection layer that maps words, sub-word units or n-grams to vector representations (often trained beforehand with unsupervised methods), and then combine them with the different architectures of neural networks, such as neural bag-of-words (NBOW), recurrent neural network (RNN) BIBREF9 , BIBREF10 , BIBREF11 and convolutional neural network (CNN) BIBREF12 , BIBREF13 .
In this paper, we use three encoders (NBOW, LSTM and attentive LSTM) to model the text descriptions.

Bag-of-Words Encoder
A simple and intuitive method is the neural bag-of-words (NBOW) model, in which the representation of text can be generated by summing up its constituent word representations.
We denote the text description as word sequence $x_{1:n} = x_1,\cdots ,x_n$ , where $x_i$ is the word at position $i$ . The NBOW encoder is 
$$\mathrm {enc_1}(x_{1:n}) = \sum _{i=1}^{n} \mathbf {x}_i,$$   (Eq. 7) 
 where $\mathbf {x}_i \in \mathbb {R}^d$ is the word embedding of $x_i$ .

Joint Structure and Text Encoder
Since both the structure and text description provide valuable information for an entity , we wish to integrate all these information into a joint representation.
We propose a united model to learn a joint representation of both structure and text information. The whole model can be end-to-end trained.
For an entity $e$ , we denote $\mathbf {e}_s$ to be its embedding of structure information, $\mathbf {e}_d$ to be encoding of its text descriptions. The main concern is how to combine $\mathbf {e}_s$ and $\mathbf {e}_d$ .
To integrate two kinds of representations of entities, we use gating mechanism to decide how much the joint representation depends on structure or text.
The joint representation $\mathbf {e}$ is a linear interpolation between the $\mathbf {e}_s$ and $\mathbf {e}_d$ . 
$$\mathbf {e}= \textbf {g}_e \odot \mathbf {e}_s + (1-\textbf {g}_e)\odot \mathbf {e}_d,$$   (Eq. 14) 
 where $\textbf {g}_e$ is a gate to balance two sources information and its elements are in $[0,1]$ , and $\odot $ is an element-wise multiplication. Intuitively, when the gate is close to 0, the joint representation is forced to ignore the structure information and is the text representation only.

Experiment
In this section, we study the empirical performance of our proposed models on two benchmark tasks: triplet classification and link prediction.

Link Prediction
Link prediction is a subtask of knowledge graph completion to complete a triplet $(h, r, t)$ with $h$ or $t$ missing, i.e., predict $t$ given $(h, r)$ or predict $h$ given $(r, t)$ . Rather than requiring one best answer, this task emphasizes more on ranking a set of candidate entities from the knowledge graph.
Similar to BIBREF2 , we use two measures as our evaluation metrics. (1) Mean Rank: the averaged rank of correct entities or relations; (2) Hits@p: the proportion of valid entities or relations ranked in top $p$ predictions. Here, we set $p=10$ for entities and $p=1$ for relations. A lower Mean Rank and a higher Hits@p should be achieved by a good embedding model. We call this evaluation setting “Raw”. Since a false predicted triplet may also exist in knowledge graphs, it should be regard as a valid triplet. Hence, we should remove the false predicted triplets included in training, validation and test sets before ranking (except the test triplet of interest). We call this evaluation setting “Filter”. The evaluation results are reported under these two settings.
We select the margin $\gamma $ among $\lbrace 1, 2\rbrace $ , the embedding dimension $d$ among $\lbrace 20, 50, 100\rbrace $ , the regularization $\eta $ among $\lbrace 0, 1E{-5}, 1E{-6}\rbrace $ , two learning rates $\lambda _s$ and $\lambda _t$ among $\lbrace 0.001, 0.01, 0.05\rbrace $ to learn the parameters of structure and text encoding. The dissimilarity measure is set to either $L_1$ or $\lbrace 1, 2\rbrace $0 distance.
In order to speed up the convergence and avoid overfitting, we initiate the structure embeddings of entity and relation with the results of TransE. The embedding of a word is initialized by averaging the linked entity embeddings whose description include this word. The rest parameters are initialized by randomly sampling from uniform distribution in $[-0.1, 0.1]$ .
The final optimal configurations are: $\gamma = 2$ , $d=20$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $\lambda _t = 0.1$ , and $L_1$ distance on WN18; $\gamma =2$ , $d=100$ , $\eta =1E{-5}$ , $\lambda _s = 0.01$ , $d=20$0 , and $d=20$1 distance on FB15K.
Experimental results on both WN18 and FB15k are shown in Table 2 , where we use “Jointly(CBOW)”, “Jointly(LSTM)” and “Jointly(A-LSTM)” to represent our jointly encoding models with CBOW, LSTM and attentive LSTM text encoders. Our baseline is TransE since that the score function of our models is based on TransE.
From the results, we observe that proposed models surpass the baseline, TransE, on all metrics, which indicates that knowledge representation can benefit greatly from text description.
On WN18, the reason why “Jointly(A-LSTM)” is slightly worse than “Jointly(LSTM)” is probably because the number of relations is limited. Therefore, the attention mechanism does not have obvious advantage. On FB15K, “Jointly(A-LSTM)” achieves the best performance and is significantly higher than baseline methods on mean rank.
Although the Hits@10 of our models are worse than the best state-of-the-art method, TransD, it is worth noticing that the score function of our models is based on TransE, not TransD. Our models are compatible with other state-of-the-art knowledge embedding models. We believe that our model can be further improved by adopting the score functions of other state-of-the-art methods, such as TransD.
Besides, textual information largely alleviates the issue of sparsity and our model achieves substantial improvement on Mean Rank comparing with TransD. However, textual information may slightly degrade the representation of frequent entities which have been well-trained. This may be another reason why our Hits@10 is worse than TransD which only utilizes structural information.
For the comparison of Hits@10 of different kinds of relations, we categorized the relationships according to the cardinalities of their head and tail arguments into four classes: 1-to-1, 1-to-many, many-to-1, many-to-many. Mapping properties of relations follows the same rules in BIBREF2 .
Table 3 shows the detailed results by mapping properties of relations on FB15k. We can see that our models outperform baseline TransE in all types of relations (1-to-1, 1-to-N, N-to-1 and N-to-N), especially when (1) predicting “1-to-1” relations and (2) predicting the 1 side for “1-to-N” and “N-to-1” relations.
To get more insights into how the joint representation is influenced by the structure and text information. We observe the activations of gates, which control the balance between two sources of information, to understand the behavior of neurons. We sort the entities by their frequencies and divide them into 50 equal-size groups of different frequencies, and average the values of all gates in each group.
Figure 3 gives the average of gates in ten groups from high- to low-frequency. We observe that the text information play more important role for the low-frequency entities.

Related Work
Recently, it has gained lots of interests to jointly learn the embeddings of knowledge graph and text information. There are several methods using textual information to help KG representation learning.
 BIBREF3 represent an entity as the average of its word embeddings in entity name, allowing the sharing of textual information located in similar entity names.
 BIBREF5 jointly embed knowledge and text into the same space by aligning the entity name and its Wikipedia anchor, which brings promising improvements to the accuracy of predicting facts. BIBREF6 extend the joint model and aligns knowledge and words in the entity descriptions. However, these two works align the two kinds of embeddings on word level, which can lose some semantic information on phrase or sentence level.
 BIBREF25 also represent entities with entity names or the average of word embeddings in descriptions. However, their use of descriptions neglects word orders, and the use of entity names struggles with ambiguity. BIBREF7 jointly learn knowledge graph embeddings with entity descriptions. They use continuous bag-of-words and convolutional neural network to encode semantics of entity descriptions. However, they separate the objective functions into two energy functions of structure-based and description-based representations. BIBREF26 embeds both entity and relation embeddings by taking KG and text into consideration using CNN. To utilize both representations, they need further estimate an optimum weight coefficients to combine them together in the specific tasks.
Besides entity representation, there are also a lot of works BIBREF27 , BIBREF28 , BIBREF29 to map textual relations and knowledge base relations to the same vector space and obtained substantial improvements.
While releasing the current paper we discovered a paper by BIBREF30 proposing a similar model with attention mechanism which is evaluated on link prediction and triplet classification. However, our work encodes text description as a whole without explicit segmentation of sentences, which breaks the order and coherence among sentences.

Conclusion
We propose a united representation for knowledge graph, utilizing both structure and text description information of the entities. Experiments show that our proposed jointly representation learning with gating mechanism is effective, which benefits to modeling the meaning of an entity.
In the future, we will consider the following research directions to improve our model:

Figure 2: Our general architecture of jointly structural and textual encoding.

Table 1: Statistics of datasets used in experiments.

Table 2: Results on link prediction.

Table 3: Detailed results by category of relationship on FB15K.

Figure 3: Detailed results by entity frequency on FB15K.

Table 4: Results on triplet classification.

Question: What baselines are used for comparison?
{"answer": ["TransE"], "evidence": ["Experimental results on both WN18 and FB15k are shown in Table 2 , where we use \u201cJointly(CBOW)\u201d, \u201cJointly(LSTM)\u201d and \u201cJointly(A-LSTM)\u201d to represent our jointly encoding models with CBOW, LSTM and attentive LSTM text encoders. Our baseline is TransE since that the score function of our models is based on TransE."]}